[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:51:39 dist-test-slave-nj21 krb5kdc[11155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:51:39 dist-test-slave-nj21 krb5kdc[11155](info): set up 2 sockets
Aug 06 23:51:39 dist-test-slave-nj21 krb5kdc[11155](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 23:51:40.945817 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:51:41 dist-test-slave-nj21 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:51:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:51:41.087160 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38107
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:38107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:41.251245 11172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:41.251561 11172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:41.251626 11172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:41.256345 11172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:51:41.256424 11172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:41.256451 11172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:41.256474 11172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:51:41.256495 11172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:51:41.262509 11172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:38107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:41.264266 11172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:41.266358 11172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:41.283082 11181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:41.283120 11177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:41.283246 11172 server_base.cc:1034] running on GCE node
W20260806 23:51:41.283087 11179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:41.284339 11172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:41.286375 11172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:41.287966 11172 hybrid_clock.cc:648] HybridClock initialized: now 1786060301287942 us; error 87 us; skew 500 ppm
Aug 06 23:51:41 dist-test-slave-nj21 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060301, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:41.291647 11172 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:51:41.292999 11172 webserver.cc:460] Webserver started at http://127.10.225.190:36497/ using document root <none> and password file <none>
I20260806 23:51:41.293972 11172 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:41.294045 11172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:41.294409 11172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:41.297454 11172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "965f75a93c7c43ffa81407ede42ebd51"
format_stamp: "Formatted at 2026-08-06 23:51:41 on dist-test-slave-nj21"
server_key: "8203965e9d24f372825fa326a0f40f27"
server_key_iv: "f0e7b2cac7e43f3ee89421553604eb56"
server_key_version: "encryptionkey@0"
I20260806 23:51:41.298023 11172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "965f75a93c7c43ffa81407ede42ebd51"
format_stamp: "Formatted at 2026-08-06 23:51:41 on dist-test-slave-nj21"
server_key: "8203965e9d24f372825fa326a0f40f27"
server_key_iv: "f0e7b2cac7e43f3ee89421553604eb56"
server_key_version: "encryptionkey@0"
I20260806 23:51:41.303830 11172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 23:51:41.308149 11191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:41.310405 11172 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 23:51:41.310617 11172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "965f75a93c7c43ffa81407ede42ebd51"
format_stamp: "Formatted at 2026-08-06 23:51:41 on dist-test-slave-nj21"
server_key: "8203965e9d24f372825fa326a0f40f27"
server_key_iv: "f0e7b2cac7e43f3ee89421553604eb56"
server_key_version: "encryptionkey@0"
I20260806 23:51:41.310781 11172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:41.333966 11172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:41.338950 11172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:41.339406 11172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:41.358497 11172 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:38107
I20260806 23:51:41.358563 11273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:38107 every 8 connection(s)
I20260806 23:51:41.360190 11172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:51:41.365840 11274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:41.369484 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11172
I20260806 23:51:41.369854 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:51:41.372583 11142 external_mini_cluster.cc:1428] Setting key a829bc74b70ed958a875890c8ade250d
I20260806 23:51:41.381194 11274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51: Bootstrap starting.
I20260806 23:51:41.388289 11274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:41.390458 11274 log.cc:824] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:41.394804 11274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51: No bootstrap required, opened a new log
I20260806 23:51:41.402885 11274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965f75a93c7c43ffa81407ede42ebd51" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38107 } }
I20260806 23:51:41.403589 11274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:41.403813 11274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965f75a93c7c43ffa81407ede42ebd51, State: Initialized, Role: FOLLOWER
I20260806 23:51:41.405078 11274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965f75a93c7c43ffa81407ede42ebd51" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38107 } }
Aug 06 23:51:41 dist-test-slave-nj21 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:41.405319 11274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:51:41.405467 11274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:51:41.405655 11274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:41.407382 11274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965f75a93c7c43ffa81407ede42ebd51" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38107 } }
I20260806 23:51:41.408178 11274 leader_election.cc:302] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 965f75a93c7c43ffa81407ede42ebd51; no voters:
I20260806 23:51:41.408694 11274 leader_election.cc:288] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:51:41.409060 11282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:41.409507 11282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [term 1 LEADER]: Becoming Leader. State: Replica: 965f75a93c7c43ffa81407ede42ebd51, State: Running, Role: LEADER
I20260806 23:51:41.410095 11282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965f75a93c7c43ffa81407ede42ebd51" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38107 } }
I20260806 23:51:41.411110 11274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:51:41.413779 11287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "965f75a93c7c43ffa81407ede42ebd51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965f75a93c7c43ffa81407ede42ebd51" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38107 } } }
I20260806 23:51:41.413949 11287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:41.414227 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:41.380172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.10.225.190:38107)
0806 23:51:41.381140 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:41.381202 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:41.382781 (+ 1579us) server_negotiation.cc:407] Connection header received
0806 23:51:41.390097 (+ 7316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:41.390129 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:41.390848 (+ 719us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:41.391329 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:41.395491 (+ 4162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:41.396842 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:41.398753 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:41.399635 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:41.399892 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:41.406481 (+ 6589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:41.406522 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:41.406540 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:41.406588 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:41.408976 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:41.409794 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:41.409799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:41.409805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:41.409980 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:41.410604 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:41.410607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:41.410609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:41.411364 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:41.411567 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:41.412645 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0806 23:51:41.412957 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":240,"threads_started":1}
I20260806 23:51:41.414556 11288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 965f75a93c7c43ffa81407ede42ebd51. Latest consensus state: current_term: 1 leader_uuid: "965f75a93c7c43ffa81407ede42ebd51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965f75a93c7c43ffa81407ede42ebd51" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38107 } } }
I20260806 23:51:41.414749 11288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51 [sys.catalog]: This master's current role is: LEADER
W20260806 23:51:41.422022 11311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:51:41.422204 11311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:51:41.422348 11300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:51:41.423779 11300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:51:41.432091 11300 catalog_manager.cc:1347] Generated new cluster ID: 3eed29bab3254f01a8b4b7cd6f139de9
I20260806 23:51:41.432181 11300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:51:41.465320 11300 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:51:41.466605 11300 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:51:41.477789 11300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 965f75a93c7c43ffa81407ede42ebd51: Generated new TSK 0
I20260806 23:51:41.478902 11300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:51:41.531231 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38107
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:41.662020 11320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:41.662339 11320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:41.662393 11320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:41.668782 11320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:41.668874 11320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:41.668942 11320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:51:41.675942 11320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:41.677346 11320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:41.678611 11320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:51:41.690429 11320 server_base.cc:1034] running on GCE node
W20260806 23:51:41.690356 11329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:41.690356 11328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:41.690358 11332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:41.691159 11320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:41.691915 11320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:41.693224 11320 hybrid_clock.cc:648] HybridClock initialized: now 1786060301693210 us; error 62 us; skew 500 ppm
Aug 06 23:51:41 dist-test-slave-nj21 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060301, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:41.696151 11320 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:51:41.697130 11320 webserver.cc:460] Webserver started at http://127.10.225.129:46675/ using document root <none> and password file <none>
I20260806 23:51:41.697715 11320 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:41.697777 11320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:41.697964 11320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:41.700183 11320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "ac2a0cea33c44ba5af945732758f2b9f"
format_stamp: "Formatted at 2026-08-06 23:51:41 on dist-test-slave-nj21"
server_key: "f11e4768fcd7c28849180d1b5d332294"
server_key_iv: "b301bf028a328f881cdd373e29054011"
server_key_version: "encryptionkey@0"
I20260806 23:51:41.700727 11320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "ac2a0cea33c44ba5af945732758f2b9f"
format_stamp: "Formatted at 2026-08-06 23:51:41 on dist-test-slave-nj21"
server_key: "f11e4768fcd7c28849180d1b5d332294"
server_key_iv: "b301bf028a328f881cdd373e29054011"
server_key_version: "encryptionkey@0"
I20260806 23:51:41.705093 11320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:51:41.708168 11343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:41.709656 11320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:51:41.709781 11320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "ac2a0cea33c44ba5af945732758f2b9f"
format_stamp: "Formatted at 2026-08-06 23:51:41 on dist-test-slave-nj21"
server_key: "f11e4768fcd7c28849180d1b5d332294"
server_key_iv: "b301bf028a328f881cdd373e29054011"
server_key_version: "encryptionkey@0"
I20260806 23:51:41.709892 11320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:41.728495 11320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:41.732589 11320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:41.732807 11320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:41.733580 11320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:51:41.734879 11320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:41.734937 11320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:41.734987 11320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:41.735013 11320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:41.745734 11320 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:43147
I20260806 23:51:41.745956 11516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:43147 every 8 connection(s)
I20260806 23:51:41.746867 11320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:51:41.749935 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11320
I20260806 23:51:41.750043 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:51:41.750345 11142 external_mini_cluster.cc:1428] Setting key db346d42d6fde8a263322731771908be
Aug 06 23:51:41 dist-test-slave-nj21 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060301, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:51:41.763913 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:41.750093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:44349 (local address 127.10.225.190:38107)
0806 23:51:41.750240 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:41.750245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:41.751452 (+ 1207us) server_negotiation.cc:407] Connection header received
0806 23:51:41.752580 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:41.752582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:41.752628 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:41.752705 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:41.754358 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:41.755248 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:41.756562 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:41.756817 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:41.756923 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:41.760043 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:41.760048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:41.760051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:41.760082 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:41.761722 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:41.762258 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:41.762262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:41.762264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:41.762366 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:41.762701 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:41.762704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:41.762706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:41.762949 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:41.763034 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:41.763583 (+ 549us) server_negotiation.cc:299] Negotiation successful
0806 23:51:41.763775 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:51:41.764674 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:41.750558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38107 (local address 127.10.225.129:44349)
0806 23:51:41.751298 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:41.751352 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:41.752342 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:41.752902 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:41.752912 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:41.753401 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:41.754075 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:41.754097 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:41.755460 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:41.755463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:41.756409 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:41.756420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:41.756974 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:41.756978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:41.757139 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:41.757942 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:41.757966 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:41.759848 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:41.761859 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:41.761865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:41.761881 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:41.762146 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:41.762454 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:41.762456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:41.762458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:41.762594 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:41.763045 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:41.763051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:41.763302 (+ 251us) client_negotiation.cc:769] Sending connection context
0806 23:51:41.763470 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 23:51:41.763781 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:51:41.766394 11517 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38107
I20260806 23:51:41.766659 11517 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:41.767549 11517 heartbeater.cc:511] Master 127.10.225.190:38107 requested a full tablet report, sending...
I20260806 23:51:41.771107 11219 ts_manager.cc:194] Registered new tserver with Master: ac2a0cea33c44ba5af945732758f2b9f (127.10.225.129:43147)
I20260806 23:51:41.772964 11219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:44349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:51:41.784675 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38107
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:41.906329 11526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:41.906622 11526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:41.906668 11526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:41.912856 11526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:41.912955 11526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:41.913031 11526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:51:41.919881 11526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:41.921087 11526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:41.922163 11526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:41.932714 11534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:41.932683 11538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:41.932719 11535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:41.933588 11526 server_base.cc:1034] running on GCE node
I20260806 23:51:41.934020 11526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:41.934706 11526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:41.935909 11526 hybrid_clock.cc:648] HybridClock initialized: now 1786060301935881 us; error 43 us; skew 500 ppm
Aug 06 23:51:41 dist-test-slave-nj21 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060301, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:41.941843 11526 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:51:41.942888 11526 webserver.cc:460] Webserver started at http://127.10.225.130:38119/ using document root <none> and password file <none>
I20260806 23:51:41.943487 11526 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:41.943558 11526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:41.943908 11526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:41.946152 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "c779166ffe254a5aa9fce313b45a3208"
format_stamp: "Formatted at 2026-08-06 23:51:41 on dist-test-slave-nj21"
server_key: "5c5facdb4b98a0d9aa3a4289118516e1"
server_key_iv: "8d89a36930fea36107a522aaedab08ec"
server_key_version: "encryptionkey@0"
I20260806 23:51:41.946735 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "c779166ffe254a5aa9fce313b45a3208"
format_stamp: "Formatted at 2026-08-06 23:51:41 on dist-test-slave-nj21"
server_key: "5c5facdb4b98a0d9aa3a4289118516e1"
server_key_iv: "8d89a36930fea36107a522aaedab08ec"
server_key_version: "encryptionkey@0"
I20260806 23:51:41.951390 11526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 23:51:41.954224 11547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:41.955595 11526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:51:41.955749 11526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "c779166ffe254a5aa9fce313b45a3208"
format_stamp: "Formatted at 2026-08-06 23:51:41 on dist-test-slave-nj21"
server_key: "5c5facdb4b98a0d9aa3a4289118516e1"
server_key_iv: "8d89a36930fea36107a522aaedab08ec"
server_key_version: "encryptionkey@0"
I20260806 23:51:41.955866 11526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:41.979617 11526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:41.983322 11526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:41.983508 11526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:41.984155 11526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:51:41.985364 11526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:41.985446 11526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:41.985502 11526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:41.985525 11526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:41.997138 11526 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:40705
I20260806 23:51:41.997291 11705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:40705 every 8 connection(s)
I20260806 23:51:41.998234 11526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:51:42.003702 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11526
I20260806 23:51:42.003854 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:51:42.004393 11142 external_mini_cluster.cc:1428] Setting key 767586f161b28af3801068a33baf3ccb
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060301, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:42.016165 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:42.000507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:41031 (local address 127.10.225.190:38107)
0806 23:51:42.000741 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:42.000747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:42.001908 (+ 1161us) server_negotiation.cc:407] Connection header received
0806 23:51:42.002803 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:42.002806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:42.002861 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:42.002941 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:42.005234 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.005950 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.007526 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.007806 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.007897 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:42.010744 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:42.010748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:42.010752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:42.010779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:42.012864 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:42.013721 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:42.013726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:42.013728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:42.013855 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:42.014505 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:42.014509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:42.014510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:42.014810 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:42.014934 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:42.015839 (+ 905us) server_negotiation.cc:299] Negotiation successful
0806 23:51:42.015986 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:51:42.017683 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:42.000818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38107 (local address 127.10.225.130:41031)
0806 23:51:42.001738 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:42.001793 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:42.002590 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:42.003364 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:42.003380 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:42.004344 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:42.005058 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:42.005081 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.006083 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.006086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:42.007380 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:42.007395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.007965 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.007969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:42.008177 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:42.008977 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:42.008995 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:42.010560 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:42.013210 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:42.013216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:42.013233 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:42.013550 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:42.014049 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:42.014055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:42.014058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:42.014359 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:42.015015 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:42.015024 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:42.015428 (+ 404us) client_negotiation.cc:769] Sending connection context
0806 23:51:42.015752 (+ 324us) client_negotiation.cc:240] Negotiation successful
0806 23:51:42.016172 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":129,"threads_started":1}
I20260806 23:51:42.019038 11707 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38107
I20260806 23:51:42.019312 11707 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:42.019946 11707 heartbeater.cc:511] Master 127.10.225.190:38107 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
I20260806 23:51:42.021240 11219 ts_manager.cc:194] Registered new tserver with Master: c779166ffe254a5aa9fce313b45a3208 (127.10.225.130:40705)
I20260806 23:51:42.021988 11219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:41031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:51:42.040344 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38107
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:42839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:42.188664 11717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:42.188977 11717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:42.189026 11717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:42.193159 11717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:42.193229 11717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:42.193291 11717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:51:42.198814 11717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:42.199978 11717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:42.201035 11717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:42.213533 11723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:42.213496 11726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:42.213687 11717 server_base.cc:1034] running on GCE node
W20260806 23:51:42.213852 11724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:42.214385 11717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:42.215152 11717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:42.216346 11717 hybrid_clock.cc:648] HybridClock initialized: now 1786060302216323 us; error 33 us; skew 500 ppm
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:42.219388 11717 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:51:42.220914 11717 webserver.cc:460] Webserver started at http://127.10.225.131:38325/ using document root <none> and password file <none>
I20260806 23:51:42.221719 11717 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:42.221807 11717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:42.222110 11717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:42.224856 11717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "14dc6393373a4f049a6f60fd26ad6c3d"
format_stamp: "Formatted at 2026-08-06 23:51:42 on dist-test-slave-nj21"
server_key: "98d26e60e5f45ff064bca4c3c4c49acd"
server_key_iv: "f776c193bbc27d069bd43bab4e9a7845"
server_key_version: "encryptionkey@0"
I20260806 23:51:42.225538 11717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "14dc6393373a4f049a6f60fd26ad6c3d"
format_stamp: "Formatted at 2026-08-06 23:51:42 on dist-test-slave-nj21"
server_key: "98d26e60e5f45ff064bca4c3c4c49acd"
server_key_iv: "f776c193bbc27d069bd43bab4e9a7845"
server_key_version: "encryptionkey@0"
I20260806 23:51:42.230922 11717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 23:51:42.234186 11736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:42.235590 11717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 23:51:42.235773 11717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "14dc6393373a4f049a6f60fd26ad6c3d"
format_stamp: "Formatted at 2026-08-06 23:51:42 on dist-test-slave-nj21"
server_key: "98d26e60e5f45ff064bca4c3c4c49acd"
server_key_iv: "f776c193bbc27d069bd43bab4e9a7845"
server_key_version: "encryptionkey@0"
I20260806 23:51:42.235908 11717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:42.257422 11717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:42.260998 11717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:42.261171 11717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:42.261749 11717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:51:42.262663 11717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:42.262712 11717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:42.262756 11717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:42.262773 11717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:42.271605 11717 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:43709
I20260806 23:51:42.271651 11886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:43709 every 8 connection(s)
I20260806 23:51:42.272614 11717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:51:42.273103 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11717
I20260806 23:51:42.273226 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:51:42.273559 11142 external_mini_cluster.cc:1428] Setting key b2f8444acfde75da4e968ee9eeeeb0e7
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:42.287395 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:42.274558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:51547 (local address 127.10.225.190:38107)
0806 23:51:42.274692 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:42.274697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:42.275842 (+ 1145us) server_negotiation.cc:407] Connection header received
0806 23:51:42.277606 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:42.277609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:42.277663 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:42.277735 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:42.279020 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.279867 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.281104 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.281355 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.281446 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:42.284119 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:42.284122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:42.284124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:42.284148 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:42.285408 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:42.285902 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:42.285905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:42.285906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:42.285982 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:42.286295 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:42.286298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:42.286300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:42.286494 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:42.286579 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:42.287093 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 23:51:42.287219 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:51:42.288480 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:42.275050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38107 (local address 127.10.225.131:51547)
0806 23:51:42.275641 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:42.275731 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:42.277380 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:42.277895 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:42.277902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:42.278288 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:42.278859 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:42.278876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.280039 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.280042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:42.280957 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:42.280967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.281522 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.281526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:42.281687 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:42.282334 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:42.282367 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:42.283951 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:42.285520 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:42.285527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:42.285543 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:42.285801 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:42.286065 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:42.286068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:42.286070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:42.286209 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:42.286594 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:42.286601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:42.286855 (+ 254us) client_negotiation.cc:769] Sending connection context
0806 23:51:42.287035 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 23:51:42.287389 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":138,"threads_started":1}
I20260806 23:51:42.289985 11889 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38107
I20260806 23:51:42.290228 11889 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:42.290743 11889 heartbeater.cc:511] Master 127.10.225.190:38107 requested a full tablet report, sending...
I20260806 23:51:42.292099 11217 ts_manager.cc:194] Registered new tserver with Master: 14dc6393373a4f049a6f60fd26ad6c3d (127.10.225.131:43709)
I20260806 23:51:42.292845 11217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:51547
I20260806 23:51:42.301937 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:51:42.333783 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:42.324349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.10.225.129:43147)
0806 23:51:42.324715 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:42.324722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:42.324751 (+ 29us) server_negotiation.cc:407] Connection header received
0806 23:51:42.324856 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:42.324864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:42.325069 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:42.325202 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:42.325915 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.326655 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.327470 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.327875 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.327971 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:42.330324 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:42.330334 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:42.330349 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:42.330381 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:42.331914 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:42.332317 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:42.332322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:42.332324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:42.332412 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:42.332660 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:42.332664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:42.332666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:42.332927 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:42.333144 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:42.333419 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 23:51:42.333592 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":123,"threads_started":1}
W20260806 23:51:42.335043 11388 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60658
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:51:42.367194 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:42.355900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.10.225.129:43147)
0806 23:51:42.356053 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:42.356058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:42.356213 (+ 155us) server_negotiation.cc:407] Connection header received
0806 23:51:42.356628 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:42.356633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:42.356718 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:42.356803 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:42.357727 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.358489 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.359514 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:42.359900 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:42.360010 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:42.363320 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:42.363324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:42.363326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:42.363367 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:42.364806 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:42.365598 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:42.365601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:42.365602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:42.365692 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:42.366064 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:42.366068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:42.366069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:42.366339 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:42.366493 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:42.366752 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 23:51:42.366899 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:51:42.368594 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11320
I20260806 23:51:42.379396 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11526
I20260806 23:51:42.400715 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11717
I20260806 23:51:42.417340 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11172
2026-08-06T23:51:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2697 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11910](info): set up 2 sockets
Aug 06 23:51:42 dist-test-slave-nj21 krb5kdc[11910](info): commencing operation
krb5kdc: starting...
W20260806 23:51:43.625576 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:51:43 dist-test-slave-nj21 krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:51:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:51:43.820302 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:46019
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:37041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:46019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:43.951285 11930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:43.951575 11930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:43.951618 11930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:43.955750 11930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:51:43.955832 11930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:43.955852 11930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:43.955868 11930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:51:43.955881 11930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:51:43.961475 11930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:46019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:46019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:43.962658 11930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:43.963775 11930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:43.972934 11940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:43.972932 11943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:43.973052 11930 server_base.cc:1034] running on GCE node
W20260806 23:51:43.972965 11945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:43.973850 11930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:43.975051 11930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:43.976233 11930 hybrid_clock.cc:648] HybridClock initialized: now 1786060303976196 us; error 56 us; skew 500 ppm
Aug 06 23:51:43 dist-test-slave-nj21 krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060303, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:43.980217 11930 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:51:43.981266 11930 webserver.cc:460] Webserver started at http://127.10.225.190:41583/ using document root <none> and password file <none>
I20260806 23:51:43.981889 11930 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:43.981962 11930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:43.982187 11930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:43.984664 11930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "66bf20befe3940f88ba494b7420999a4"
format_stamp: "Formatted at 2026-08-06 23:51:43 on dist-test-slave-nj21"
server_key: "f6608f34726b4dabaeb8c7daeba5d52a"
server_key_iv: "42f8660333ef4367d8fa7abacf65a51f"
server_key_version: "encryptionkey@0"
I20260806 23:51:43.985281 11930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "66bf20befe3940f88ba494b7420999a4"
format_stamp: "Formatted at 2026-08-06 23:51:43 on dist-test-slave-nj21"
server_key: "f6608f34726b4dabaeb8c7daeba5d52a"
server_key_iv: "42f8660333ef4367d8fa7abacf65a51f"
server_key_version: "encryptionkey@0"
I20260806 23:51:43.989581 11930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 23:51:43.992864 11953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:43.994837 11930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:51:43.995085 11930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "66bf20befe3940f88ba494b7420999a4"
format_stamp: "Formatted at 2026-08-06 23:51:43 on dist-test-slave-nj21"
server_key: "f6608f34726b4dabaeb8c7daeba5d52a"
server_key_iv: "42f8660333ef4367d8fa7abacf65a51f"
server_key_version: "encryptionkey@0"
I20260806 23:51:43.995267 11930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:44.014571 11930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:44.019361 11930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:44.019742 11930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:44.028723 11930 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:46019
I20260806 23:51:44.028749 12023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:46019 every 8 connection(s)
I20260806 23:51:44.029840 11930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:51:44.030725 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11930
I20260806 23:51:44.030844 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:51:44.031131 11142 external_mini_cluster.cc:1428] Setting key dc4aa51e584167818492edf0c18fff00
I20260806 23:51:44.033466 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:44.039316 12024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4: Bootstrap starting.
Aug 06 23:51:44 dist-test-slave-nj21 krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:44.041918 12024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:44.042711 12024 log.cc:824] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:44.044844 12024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4: No bootstrap required, opened a new log
I20260806 23:51:44.046622 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.032727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.10.225.190:46019)
0806 23:51:44.033225 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:44.033242 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:44.033295 (+ 53us) server_negotiation.cc:407] Connection header received
0806 23:51:44.034022 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:44.034041 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:44.034376 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:44.034704 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:44.035505 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.036553 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.037380 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.037841 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.037984 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.040527 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:44.040533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:44.040547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:44.040575 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:44.042697 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.043139 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.043144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.043150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.043266 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.043940 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.043945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.043947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.044758 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:44.044991 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:44.045319 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 23:51:44.045632 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":153,"threads_started":1}
I20260806 23:51:44.048166 12024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bf20befe3940f88ba494b7420999a4" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46019 } }
I20260806 23:51:44.048466 12024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:44.048520 12024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bf20befe3940f88ba494b7420999a4, State: Initialized, Role: FOLLOWER
I20260806 23:51:44.048997 12024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bf20befe3940f88ba494b7420999a4" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46019 } }
I20260806 23:51:44.049139 12024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:51:44.049201 12024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:51:44.049288 12024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:44.050294 12024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bf20befe3940f88ba494b7420999a4" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46019 } }
I20260806 23:51:44.050634 12024 leader_election.cc:302] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66bf20befe3940f88ba494b7420999a4; no voters:
I20260806 23:51:44.050961 12024 leader_election.cc:288] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:51:44.051084 12030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:44.051326 12030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [term 1 LEADER]: Becoming Leader. State: Replica: 66bf20befe3940f88ba494b7420999a4, State: Running, Role: LEADER
I20260806 23:51:44.051645 12030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bf20befe3940f88ba494b7420999a4" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46019 } }
I20260806 23:51:44.052278 12024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:51:44.053555 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66bf20befe3940f88ba494b7420999a4. Latest consensus state: current_term: 1 leader_uuid: "66bf20befe3940f88ba494b7420999a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bf20befe3940f88ba494b7420999a4" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46019 } } }
I20260806 23:51:44.053718 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:44.053545 12031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66bf20befe3940f88ba494b7420999a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bf20befe3940f88ba494b7420999a4" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46019 } } }
I20260806 23:51:44.053869 12031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4 [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:44.054198 12041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:51:44.057797 12041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:51:44.065503 12041 catalog_manager.cc:1347] Generated new cluster ID: 17d83a7bf5554b2086be0325d819fc54
I20260806 23:51:44.065604 12041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:51:44.089637 12041 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:51:44.091056 12041 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:51:44.102303 12041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66bf20befe3940f88ba494b7420999a4: Generated new TSK 0
I20260806 23:51:44.103385 12041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:51:44.143652 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46019
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:37041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:44.283250 12062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:44.283574 12062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:44.283624 12062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:44.288388 12062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:44.288476 12062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:44.288544 12062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:51:44.295085 12062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:44.296808 12062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:44.298192 12062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:44.309480 12073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:44.309509 12078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:44.309760 12062 server_base.cc:1034] running on GCE node
W20260806 23:51:44.309613 12070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:44.310410 12062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:44.311210 12062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:44.312440 12062 hybrid_clock.cc:648] HybridClock initialized: now 1786060304312407 us; error 45 us; skew 500 ppm
Aug 06 23:51:44 dist-test-slave-nj21 krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:44.316166 12062 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:51:44.317452 12062 webserver.cc:460] Webserver started at http://127.10.225.129:39323/ using document root <none> and password file <none>
I20260806 23:51:44.318584 12062 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:44.318694 12062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:44.318970 12062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:44.322158 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "05c847bb4b304431ba5a0afed5146809"
format_stamp: "Formatted at 2026-08-06 23:51:44 on dist-test-slave-nj21"
server_key: "f06d2ac1560c50aa62b8ae7dd4183065"
server_key_iv: "4c48206de715392065d78ab2df3af5f2"
server_key_version: "encryptionkey@0"
I20260806 23:51:44.322841 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "05c847bb4b304431ba5a0afed5146809"
format_stamp: "Formatted at 2026-08-06 23:51:44 on dist-test-slave-nj21"
server_key: "f06d2ac1560c50aa62b8ae7dd4183065"
server_key_iv: "4c48206de715392065d78ab2df3af5f2"
server_key_version: "encryptionkey@0"
I20260806 23:51:44.328052 12062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 23:51:44.330942 12090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:44.332432 12062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:51:44.332554 12062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "05c847bb4b304431ba5a0afed5146809"
format_stamp: "Formatted at 2026-08-06 23:51:44 on dist-test-slave-nj21"
server_key: "f06d2ac1560c50aa62b8ae7dd4183065"
server_key_iv: "4c48206de715392065d78ab2df3af5f2"
server_key_version: "encryptionkey@0"
I20260806 23:51:44.332685 12062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:44.355232 12062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:44.358609 12062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:44.358800 12062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:44.359444 12062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:51:44.360352 12062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:44.360406 12062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:44.360451 12062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:44.360479 12062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:44.370107 12062 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:44251
I20260806 23:51:44.370172 12255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:44251 every 8 connection(s)
I20260806 23:51:44.371186 12062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:51:44.372745 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 12062
I20260806 23:51:44.372857 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:51:44.373148 11142 external_mini_cluster.cc:1428] Setting key da4700eb7c267a8048928457fe321a4f
Aug 06 23:51:44 dist-test-slave-nj21 krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:51:44.386662 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.373412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:49663 (local address 127.10.225.190:46019)
0806 23:51:44.373596 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:44.373602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:44.374649 (+ 1047us) server_negotiation.cc:407] Connection header received
0806 23:51:44.376112 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:44.376116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:44.376171 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:44.376244 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:44.377852 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.378533 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.379808 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.380062 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.380157 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.382927 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:44.382931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:44.382933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:44.382964 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:44.384574 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.385079 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.385083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.385085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.385162 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.385489 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.385492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.385494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.385693 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:44.385782 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:44.386329 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 23:51:44.386477 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:51:44.387506 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.373751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46019 (local address 127.10.225.129:49663)
0806 23:51:44.374464 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:44.374522 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:44.375898 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:44.376437 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:44.376446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:44.376963 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:44.377675 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.377695 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.378703 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.378706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.379624 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.379635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.380220 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.380225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.380398 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.381031 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:44.381049 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:44.382753 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:44.384692 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:44.384698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:44.384716 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:44.384980 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:44.385246 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:44.385249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:44.385251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:44.385395 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:44.385804 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:44.385811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:44.386082 (+ 271us) client_negotiation.cc:769] Sending connection context
0806 23:51:44.386261 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 23:51:44.386616 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":165,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:51:44.388962 12257 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46019
I20260806 23:51:44.389230 12257 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:44.389866 12257 heartbeater.cc:511] Master 127.10.225.190:46019 requested a full tablet report, sending...
I20260806 23:51:44.392004 11970 ts_manager.cc:194] Registered new tserver with Master: 05c847bb4b304431ba5a0afed5146809 (127.10.225.129:44251)
I20260806 23:51:44.393194 11970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:49663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:51:44.406970 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46019
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:37041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:44.545251 12263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:44.545584 12263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:44.545641 12263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:44.551087 12263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:44.551172 12263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:44.551242 12263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:51:44.557354 12263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:44.558636 12263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:44.560194 12263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:44.571914 12272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:44.571923 12273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:44.571913 12276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:44.572131 12263 server_base.cc:1034] running on GCE node
I20260806 23:51:44.572587 12263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:44.573293 12263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:44.574492 12263 hybrid_clock.cc:648] HybridClock initialized: now 1786060304574471 us; error 33 us; skew 500 ppm
Aug 06 23:51:44 dist-test-slave-nj21 krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:44.578088 12263 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:51:44.579149 12263 webserver.cc:460] Webserver started at http://127.10.225.130:37441/ using document root <none> and password file <none>
I20260806 23:51:44.579847 12263 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:44.579924 12263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:44.580140 12263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:44.582455 12263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "2d2b843e423f4d0197d9eaf965d1f643"
format_stamp: "Formatted at 2026-08-06 23:51:44 on dist-test-slave-nj21"
server_key: "de3213ec78c0a389c614848f940c14f0"
server_key_iv: "4287211e215583834875e46da9691d17"
server_key_version: "encryptionkey@0"
I20260806 23:51:44.583036 12263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "2d2b843e423f4d0197d9eaf965d1f643"
format_stamp: "Formatted at 2026-08-06 23:51:44 on dist-test-slave-nj21"
server_key: "de3213ec78c0a389c614848f940c14f0"
server_key_iv: "4287211e215583834875e46da9691d17"
server_key_version: "encryptionkey@0"
I20260806 23:51:44.587092 12263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260806 23:51:44.589828 12285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:44.591750 12263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:51:44.591883 12263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "2d2b843e423f4d0197d9eaf965d1f643"
format_stamp: "Formatted at 2026-08-06 23:51:44 on dist-test-slave-nj21"
server_key: "de3213ec78c0a389c614848f940c14f0"
server_key_iv: "4287211e215583834875e46da9691d17"
server_key_version: "encryptionkey@0"
I20260806 23:51:44.591995 12263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:44.612317 12263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:44.616611 12263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:44.616813 12263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:44.617687 12263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:51:44.619053 12263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:44.619113 12263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:44.619223 12263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:44.619248 12263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:44.629809 12263 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:38971
I20260806 23:51:44.629889 12452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:38971 every 8 connection(s)
I20260806 23:51:44.630890 12263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:51:44.637221 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 12263
I20260806 23:51:44.637396 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:51:44.637706 11142 external_mini_cluster.cc:1428] Setting key f41839c652ea89a3ec3eaea5be263eda
Aug 06 23:51:44 dist-test-slave-nj21 krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:44.648950 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.633013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:44633 (local address 127.10.225.190:46019)
0806 23:51:44.633159 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:44.633163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:44.634639 (+ 1476us) server_negotiation.cc:407] Connection header received
0806 23:51:44.636424 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:44.636428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:44.636516 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:44.636596 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:44.638427 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.639041 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.640712 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.640980 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.641087 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.644675 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:44.644679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:44.644681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:44.644719 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:44.646221 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.647006 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.647009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.647011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.647096 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.647500 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.647503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.647504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.647807 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:44.647888 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:44.648647 (+ 759us) server_negotiation.cc:299] Negotiation successful
0806 23:51:44.648769 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:51:44.650056 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.633564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46019 (local address 127.10.225.130:44633)
0806 23:51:44.634359 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:44.634423 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:44.636081 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:44.636870 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:44.636882 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:44.637403 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:44.638127 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.638151 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.639226 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.639230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.640542 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.640559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.641167 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.641172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.641375 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.642166 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:44.642188 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:44.644381 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:44.646380 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:44.646389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:44.646411 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:44.646850 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:44.647183 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:44.647187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:44.647189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:44.647400 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:44.647910 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:44.647919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:44.648318 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 23:51:44.648563 (+ 245us) client_negotiation.cc:240] Negotiation successful
0806 23:51:44.648994 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":196,"threads_started":1}
I20260806 23:51:44.651603 12453 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46019
I20260806 23:51:44.651845 12453 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:44.652403 12453 heartbeater.cc:511] Master 127.10.225.190:46019 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
I20260806 23:51:44.653700 11970 ts_manager.cc:194] Registered new tserver with Master: 2d2b843e423f4d0197d9eaf965d1f643 (127.10.225.130:38971)
I20260806 23:51:44.654502 11970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:44633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:51:44.671309 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46019
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:37041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:44.802685 12462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:44.802994 12462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:44.803061 12462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:44.807650 12462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:44.807760 12462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:44.807850 12462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:51:44.813717 12462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:44.814944 12462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:44.816476 12462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:44.829496 12469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:44.829748 12462 server_base.cc:1034] running on GCE node
W20260806 23:51:44.829514 12471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:44.829514 12468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:44.830489 12462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:44.831218 12462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:44.832445 12462 hybrid_clock.cc:648] HybridClock initialized: now 1786060304832418 us; error 34 us; skew 500 ppm
Aug 06 23:51:44 dist-test-slave-nj21 krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:44.840633 12462 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:51:44.841691 12462 webserver.cc:460] Webserver started at http://127.10.225.131:41681/ using document root <none> and password file <none>
I20260806 23:51:44.842294 12462 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:44.842558 12462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:44.842949 12462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:44.845837 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "6491c172bcdf4bfca780d80c95c1d6ba"
format_stamp: "Formatted at 2026-08-06 23:51:44 on dist-test-slave-nj21"
server_key: "8c76ef846509639811efa96d2584dfdc"
server_key_iv: "abbd5ae5edcf93056dbb6e964697413c"
server_key_version: "encryptionkey@0"
I20260806 23:51:44.846488 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "6491c172bcdf4bfca780d80c95c1d6ba"
format_stamp: "Formatted at 2026-08-06 23:51:44 on dist-test-slave-nj21"
server_key: "8c76ef846509639811efa96d2584dfdc"
server_key_iv: "abbd5ae5edcf93056dbb6e964697413c"
server_key_version: "encryptionkey@0"
I20260806 23:51:44.851213 12462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:51:44.854813 12482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:44.856468 12462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:51:44.856621 12462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "6491c172bcdf4bfca780d80c95c1d6ba"
format_stamp: "Formatted at 2026-08-06 23:51:44 on dist-test-slave-nj21"
server_key: "8c76ef846509639811efa96d2584dfdc"
server_key_iv: "abbd5ae5edcf93056dbb6e964697413c"
server_key_version: "encryptionkey@0"
I20260806 23:51:44.856736 12462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:44.869508 12462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:44.872895 12462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:44.873098 12462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:44.873744 12462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:51:44.874670 12462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:44.874717 12462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:44.874771 12462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:44.874801 12462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:44.884124 12462 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:35589
I20260806 23:51:44.884157 12654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:35589 every 8 connection(s)
I20260806 23:51:44.885058 12462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:51:44.890923 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 12462
I20260806 23:51:44.891038 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:51:44.891312 11142 external_mini_cluster.cc:1428] Setting key a65cc5ae4f2349b23bc583470faef5f6
Aug 06 23:51:44 dist-test-slave-nj21 krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060304, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:44.901451 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.887032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:55531 (local address 127.10.225.190:46019)
0806 23:51:44.887179 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:44.887183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:44.888217 (+ 1034us) server_negotiation.cc:407] Connection header received
0806 23:51:44.889531 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:44.889535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:44.889600 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:44.889679 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:44.890955 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.891579 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.892906 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.893176 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.893266 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.896840 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:44.896844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:44.896846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:44.896874 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:44.898512 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.899410 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.899414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.899416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.899524 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.900005 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.900009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.900011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.900262 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:44.900412 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:44.901111 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 23:51:44.901263 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:51:44.902812 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.887419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46019 (local address 127.10.225.131:55531)
0806 23:51:44.888032 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:44.888085 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:44.889296 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:44.889871 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:44.889879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:44.890279 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:44.890802 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.890816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.891726 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.891729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.892784 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.892797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.893366 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.893370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.893576 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.894420 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:44.894446 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:44.896650 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:44.898706 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:44.898716 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:44.898739 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:44.899220 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:44.899614 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:44.899617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:44.899619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:44.899852 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:44.900476 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:44.900486 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:44.900814 (+ 328us) client_negotiation.cc:769] Sending connection context
0806 23:51:44.901030 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 23:51:44.901423 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":131,"threads_started":1}
I20260806 23:51:44.904279 12655 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46019
I20260806 23:51:44.904572 12655 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:44.905190 12655 heartbeater.cc:511] Master 127.10.225.190:46019 requested a full tablet report, sending...
I20260806 23:51:44.906421 11970 ts_manager.cc:194] Registered new tserver with Master: 6491c172bcdf4bfca780d80c95c1d6ba (127.10.225.131:35589)
I20260806 23:51:44.906975 11970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:55531
I20260806 23:51:44.918984 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:51:44.933073 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.925129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.10.225.190:46019)
0806 23:51:44.925366 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:44.925371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:44.925475 (+ 104us) server_negotiation.cc:407] Connection header received
0806 23:51:44.925637 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:44.925640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:44.925688 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:44.925760 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:44.926518 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.927037 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.927951 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.928164 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.928251 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.929662 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:44.929666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:44.929667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:44.929692 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:44.931250 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.931699 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.931702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.931704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.931786 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:44.932033 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:44.932036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:44.932038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:44.932278 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:44.932402 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:44.932773 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 23:51:44.932902 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260806 23:51:44.941883 11970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:51:44.945669 11970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:51:44.963706 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.958101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.10.225.129:44251)
0806 23:51:44.958889 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:44.958896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:44.958917 (+ 21us) server_negotiation.cc:407] Connection header received
0806 23:51:44.958992 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:44.958997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:44.959165 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:44.959292 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:44.959725 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.960521 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.962400 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.962967 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.963091 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.963325 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:44.963411 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 23:51:44.963514 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":190,"threads_started":1}
I20260806 23:51:44.963698 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.957244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:44251 (local address 127.0.0.1:48232)
0806 23:51:44.958572 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:44.958584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:44.958656 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:44.959436 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:44.959439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:44.959457 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:44.959590 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.959596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.960700 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.960703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.962297 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.962306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.963085 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.963089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.963210 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.963253 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 23:51:44.963381 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 23:51:44.963515 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"mutex_wait_us":113,"thread_start_us":128,"threads_started":1}
I20260806 23:51:44.963927 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.957091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:38971 (local address 127.0.0.1:37292)
0806 23:51:44.958137 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:44.958198 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:44.958349 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:44.959406 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:44.959409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:44.959425 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:44.959743 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.959751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.961066 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.961069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.962095 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.962105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.963416 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.963420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.963563 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.963609 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 23:51:44.963706 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 23:51:44.963781 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":253,"spinlock_wait_cycles":2816,"thread_start_us":200,"threads_started":1}
I20260806 23:51:44.963994 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.958027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.10.225.131:35589)
0806 23:51:44.958504 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:44.958517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:44.958541 (+ 24us) server_negotiation.cc:407] Connection header received
0806 23:51:44.958624 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:44.958629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:44.958864 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:44.959023 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:44.959508 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.960722 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.962366 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.963138 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.963237 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.963346 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:44.963756 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 23:51:44.963860 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":141,"threads_started":1}
I20260806 23:51:44.964323 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.957078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:35589 (local address 127.0.0.1:39320)
0806 23:51:44.958139 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:44.958209 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:44.958351 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:44.959217 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:44.959223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:44.959246 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:44.959390 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.959397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.960843 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.960847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.962224 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:44.962237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.963416 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.963420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:44.963559 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.963613 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 23:51:44.964119 (+ 506us) client_negotiation.cc:240] Negotiation successful
0806 23:51:44.964196 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":92,"thread_start_us":113,"threads_started":1}
I20260806 23:51:44.964457 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:44.957530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.10.225.130:38971)
0806 23:51:44.958880 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:44.958894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:44.958919 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:51:44.959000 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:44.959005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:44.959207 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:44.959568 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:44.959859 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.960950 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.962227 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:44.963081 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:44.963213 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:44.963356 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:44.964204 (+ 848us) server_negotiation.cc:299] Negotiation successful
0806 23:51:44.964306 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1203,"thread_start_us":152,"threads_started":1}
I20260806 23:51:44.967463 12155 tablet_service.cc:1467] Processing CreateTablet for tablet 49644e61c8fc4bea9b2bc8d0ba5f30ad (DEFAULT_TABLE table=test-table [id=e13c750d22f145a18317c2567315a8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:44.968963 12155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49644e61c8fc4bea9b2bc8d0ba5f30ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:44.969408 12552 tablet_service.cc:1467] Processing CreateTablet for tablet 49644e61c8fc4bea9b2bc8d0ba5f30ad (DEFAULT_TABLE table=test-table [id=e13c750d22f145a18317c2567315a8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:44.969576 12353 tablet_service.cc:1467] Processing CreateTablet for tablet 49644e61c8fc4bea9b2bc8d0ba5f30ad (DEFAULT_TABLE table=test-table [id=e13c750d22f145a18317c2567315a8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:44.970508 12353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49644e61c8fc4bea9b2bc8d0ba5f30ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:44.970528 12552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49644e61c8fc4bea9b2bc8d0ba5f30ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:44.977650 12679 tablet_bootstrap.cc:492] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba: Bootstrap starting.
I20260806 23:51:44.978295 12678 tablet_bootstrap.cc:492] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809: Bootstrap starting.
I20260806 23:51:44.980423 12680 tablet_bootstrap.cc:492] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643: Bootstrap starting.
I20260806 23:51:44.981989 12679 tablet_bootstrap.cc:654] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:44.982792 12680 tablet_bootstrap.cc:654] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:44.983500 12679 log.cc:824] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:44.984114 12680 log.cc:824] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:44.985419 12678 tablet_bootstrap.cc:654] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:44.985882 12679 tablet_bootstrap.cc:492] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba: No bootstrap required, opened a new log
I20260806 23:51:44.986095 12679 ts_tablet_manager.cc:1397] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260806 23:51:44.986433 12678 log.cc:824] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:44.986553 12680 tablet_bootstrap.cc:492] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643: No bootstrap required, opened a new log
I20260806 23:51:44.986806 12680 ts_tablet_manager.cc:1397] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 23:51:44.989065 12678 tablet_bootstrap.cc:492] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809: No bootstrap required, opened a new log
I20260806 23:51:44.989279 12678 ts_tablet_manager.cc:1397] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260806 23:51:44.990098 12680 raft_consensus.cc:348] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } }
I20260806 23:51:44.990075 12679 raft_consensus.cc:348] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } }
I20260806 23:51:44.990350 12680 raft_consensus.cc:374] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:44.990350 12679 raft_consensus.cc:374] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:44.990404 12680 raft_consensus.cc:729] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2b843e423f4d0197d9eaf965d1f643, State: Initialized, Role: FOLLOWER
I20260806 23:51:44.990404 12679 raft_consensus.cc:729] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6491c172bcdf4bfca780d80c95c1d6ba, State: Initialized, Role: FOLLOWER
I20260806 23:51:44.990852 12680 consensus_queue.cc:260] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } }
I20260806 23:51:44.990867 12679 consensus_queue.cc:260] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } }
I20260806 23:51:44.991657 12453 heartbeater.cc:503] Master 127.10.225.190:46019 was elected leader, sending a full tablet report...
I20260806 23:51:44.991982 12679 ts_tablet_manager.cc:1428] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:51:44.992015 12680 ts_tablet_manager.cc:1428] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:51:44.992316 12655 heartbeater.cc:503] Master 127.10.225.190:46019 was elected leader, sending a full tablet report...
I20260806 23:51:44.994092 12678 raft_consensus.cc:348] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } }
I20260806 23:51:44.994326 12678 raft_consensus.cc:374] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:44.994381 12678 raft_consensus.cc:729] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c847bb4b304431ba5a0afed5146809, State: Initialized, Role: FOLLOWER
I20260806 23:51:44.994803 12678 consensus_queue.cc:260] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } }
I20260806 23:51:44.995417 12257 heartbeater.cc:503] Master 127.10.225.190:46019 was elected leader, sending a full tablet report...
I20260806 23:51:44.996011 12678 ts_tablet_manager.cc:1428] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 23:51:45.060386 12686 raft_consensus.cc:482] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:51:45.060752 12686 raft_consensus.cc:504] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } }
I20260806 23:51:45.062409 12686 leader_election.cc:288] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d2b843e423f4d0197d9eaf965d1f643 (127.10.225.130:38971), 05c847bb4b304431ba5a0afed5146809 (127.10.225.129:44251)
I20260806 23:51:45.068471 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:45.062938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:52047 (local address 127.10.225.130:38971)
0806 23:51:45.063103 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:45.063109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:45.063237 (+ 128us) server_negotiation.cc:407] Connection header received
0806 23:51:45.063383 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:45.063386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:45.063449 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:45.063549 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:45.063952 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.064771 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.066756 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.067636 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.067816 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:45.067883 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:45.068229 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 23:51:45.068328 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 23:51:45.068471 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:45.063006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:35161 (local address 127.10.225.129:44251)
0806 23:51:45.063154 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:45.063159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:45.063486 (+ 327us) server_negotiation.cc:407] Connection header received
0806 23:51:45.063553 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:45.063556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:45.063600 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:45.063721 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:45.063982 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.064752 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.066756 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.067643 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.067816 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:45.067883 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:45.068226 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 23:51:45.068328 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 23:51:45.068472 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:45.062853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:38971 (local address 127.10.225.131:52047)
0806 23:51:45.063101 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:45.063119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:45.063257 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:45.063582 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:45.063584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:45.063600 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:45.063802 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:45.063810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.064938 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.064944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:45.066469 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:45.066486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.067830 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.067834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:45.068016 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:45.068076 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 23:51:45.068163 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 23:51:45.068290 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":182}
I20260806 23:51:45.068480 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:45.062918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:44251 (local address 127.10.225.131:35161)
0806 23:51:45.063372 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:45.063386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:45.063483 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:45.063719 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:45.063722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:45.063742 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:45.063873 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:45.063880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.064938 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.064944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:45.066469 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:45.066486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.067830 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.067834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:45.067999 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:45.068076 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 23:51:45.068163 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 23:51:45.068284 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":191,"thread_start_us":147,"threads_started":1}
I20260806 23:51:45.069371 12193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49644e61c8fc4bea9b2bc8d0ba5f30ad" candidate_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05c847bb4b304431ba5a0afed5146809" is_pre_election: true
I20260806 23:51:45.069387 12383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49644e61c8fc4bea9b2bc8d0ba5f30ad" candidate_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2b843e423f4d0197d9eaf965d1f643" is_pre_election: true
I20260806 23:51:45.069860 12383 raft_consensus.cc:2454] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6491c172bcdf4bfca780d80c95c1d6ba in term 0.
I20260806 23:51:45.070500 12489 leader_election.cc:302] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2b843e423f4d0197d9eaf965d1f643, 6491c172bcdf4bfca780d80c95c1d6ba; no voters:
I20260806 23:51:45.070801 12686 raft_consensus.cc:2781] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:51:45.070904 12686 raft_consensus.cc:482] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:51:45.070957 12686 raft_consensus.cc:3037] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:45.072127 12193 raft_consensus.cc:2454] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6491c172bcdf4bfca780d80c95c1d6ba in term 0.
I20260806 23:51:45.072168 12686 raft_consensus.cc:504] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } }
I20260806 23:51:45.072615 12686 leader_election.cc:288] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [CANDIDATE]: Term 1 election: Requested vote from peers 2d2b843e423f4d0197d9eaf965d1f643 (127.10.225.130:38971), 05c847bb4b304431ba5a0afed5146809 (127.10.225.129:44251)
I20260806 23:51:45.073032 12383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49644e61c8fc4bea9b2bc8d0ba5f30ad" candidate_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2b843e423f4d0197d9eaf965d1f643"
I20260806 23:51:45.073096 12193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49644e61c8fc4bea9b2bc8d0ba5f30ad" candidate_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05c847bb4b304431ba5a0afed5146809"
I20260806 23:51:45.073172 12383 raft_consensus.cc:3037] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:45.073199 12193 raft_consensus.cc:3037] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:45.074536 12193 raft_consensus.cc:2454] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6491c172bcdf4bfca780d80c95c1d6ba in term 1.
I20260806 23:51:45.074536 12383 raft_consensus.cc:2454] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6491c172bcdf4bfca780d80c95c1d6ba in term 1.
I20260806 23:51:45.074996 12489 leader_election.cc:302] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2b843e423f4d0197d9eaf965d1f643, 6491c172bcdf4bfca780d80c95c1d6ba; no voters:
I20260806 23:51:45.075361 12686 raft_consensus.cc:2781] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:45.075763 12686 raft_consensus.cc:686] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [term 1 LEADER]: Becoming Leader. State: Replica: 6491c172bcdf4bfca780d80c95c1d6ba, State: Running, Role: LEADER
I20260806 23:51:45.076242 12686 consensus_queue.cc:237] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } }
I20260806 23:51:45.080623 11972 catalog_manager.cc:5495] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 6491c172bcdf4bfca780d80c95c1d6ba (127.10.225.131). New cstate: current_term: 1 leader_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2b843e423f4d0197d9eaf965d1f643" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6491c172bcdf4bfca780d80c95c1d6ba" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05c847bb4b304431ba5a0afed5146809" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 44251 } health_report { overall_health: UNKNOWN } } }
Aug 06 23:51:45 dist-test-slave-nj21 krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
W20260806 23:51:45.122913 12258 tablet.cc:2369] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 05c847bb4b304431ba5a0afed5146809: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:51:45.125087 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:45.114481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.10.225.129:44251)
0806 23:51:45.114746 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:45.114751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:45.114856 (+ 105us) server_negotiation.cc:407] Connection header received
0806 23:51:45.114994 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:45.114997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:45.115045 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:45.115112 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:45.115854 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.116531 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.117639 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.117875 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.117976 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:45.121274 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:45.121291 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:45.121313 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:45.121350 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:45.123187 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:45.123646 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:45.123651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:45.123653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:45.123798 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:45.124052 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:45.124055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:45.124057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:45.124366 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:45.124568 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:45.124766 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 23:51:45.124901 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
W20260806 23:51:45.132125 12454 tablet.cc:2369] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 2d2b843e423f4d0197d9eaf965d1f643: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:51:45.136096 12656 tablet.cc:2369] T 49644e61c8fc4bea9b2bc8d0ba5f30ad P 6491c172bcdf4bfca780d80c95c1d6ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 23:51:45 dist-test-slave-nj21 krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:51:45 dist-test-slave-nj21 krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:51:45.157039 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:45.146443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.10.225.129:44251)
0806 23:51:45.146656 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:45.146662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:45.146891 (+ 229us) server_negotiation.cc:407] Connection header received
0806 23:51:45.147042 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:45.147044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:45.147093 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:45.147164 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:45.147925 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.148455 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.149267 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.149515 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.149605 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:45.152778 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:45.152782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:45.152784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:45.152813 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:45.154615 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:45.155231 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:45.155234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:45.155235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:45.155333 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:45.155777 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:45.155782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:45.155783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:45.156057 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:45.156186 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:45.156660 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 23:51:45.156858 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Aug 06 23:51:45 dist-test-slave-nj21 krb5kdc[11910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:51:45 dist-test-slave-nj21 krb5kdc[11910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:51:45.190217 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:45.180559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.10.225.129:44251)
0806 23:51:45.180801 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:45.180807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:45.180985 (+ 178us) server_negotiation.cc:407] Connection header received
0806 23:51:45.181135 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:45.181138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:45.181186 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:45.181256 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:45.182412 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.182975 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.183974 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:45.184208 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:45.184305 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:45.186988 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:45.186993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:45.186995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:45.187036 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:45.188418 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:45.188896 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:45.188900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:45.188901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:45.188990 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:45.189350 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:45.189353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:45.189354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:45.189581 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:45.189688 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:45.189965 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 23:51:45.190078 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260806 23:51:45.192116 12128 tablet_service.cc:2908] Rejecting scan request for tablet 49644e61c8fc4bea9b2bc8d0ba5f30ad: Uninitialized: clean time has not yet been initialized
I20260806 23:51:45.194796 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 12062
I20260806 23:51:45.210440 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 12263
I20260806 23:51:45.225013 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 12462
I20260806 23:51:45.246949 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11930
2026-08-06T23:51:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2829 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:51:45 dist-test-slave-nj21 krb5kdc[12711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:51:45 dist-test-slave-nj21 krb5kdc[12711](info): set up 2 sockets
Aug 06 23:51:45 dist-test-slave-nj21 krb5kdc[12711](info): commencing operation
krb5kdc: starting...
W20260806 23:51:46.430899 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:51:46 dist-test-slave-nj21 krb5kdc[12711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:51:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:51:46.578759 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38125
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:38125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:46.702039 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:46.702358 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:46.702419 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:46.706594 12726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:51:46.706658 12726 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 23:51:46.706683 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:46.706707 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:46.706729 12726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:51:46.706758 12726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:51:46.712378 12726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:38125
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:46.713548 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:46.714653 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:46.723989 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:46.724009 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:46.723989 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:46.724413 12726 server_base.cc:1034] running on GCE node
I20260806 23:51:46.725097 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:46.726344 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:46.727526 12726 hybrid_clock.cc:648] HybridClock initialized: now 1786060306727486 us; error 52 us; skew 500 ppm
Aug 06 23:51:46 dist-test-slave-nj21 krb5kdc[12711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060306, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:46.731376 12726 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:51:46.732477 12726 webserver.cc:460] Webserver started at http://127.10.225.190:42621/ using document root <none> and password file <none>
I20260806 23:51:46.733093 12726 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:46.733165 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:46.733389 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:46.735596 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "14a2989147ea4d4a8ac9af0e8eac42bc"
format_stamp: "Formatted at 2026-08-06 23:51:46 on dist-test-slave-nj21"
server_key: "a8dd86e028c014e7c87479f89d128f16"
server_key_iv: "105cbf1a46b43815218119a9c7485268"
server_key_version: "encryptionkey@0"
I20260806 23:51:46.736225 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "14a2989147ea4d4a8ac9af0e8eac42bc"
format_stamp: "Formatted at 2026-08-06 23:51:46 on dist-test-slave-nj21"
server_key: "a8dd86e028c014e7c87479f89d128f16"
server_key_iv: "105cbf1a46b43815218119a9c7485268"
server_key_version: "encryptionkey@0"
I20260806 23:51:46.740772 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:51:46.743731 12748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:46.745149 12726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:51:46.745273 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "14a2989147ea4d4a8ac9af0e8eac42bc"
format_stamp: "Formatted at 2026-08-06 23:51:46 on dist-test-slave-nj21"
server_key: "a8dd86e028c014e7c87479f89d128f16"
server_key_iv: "105cbf1a46b43815218119a9c7485268"
server_key_version: "encryptionkey@0"
I20260806 23:51:46.745405 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:46.764683 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:46.768205 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:46.768501 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:46.777242 12726 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:38125
I20260806 23:51:46.777390 12842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:38125 every 8 connection(s)
I20260806 23:51:46.778363 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:51:46.781337 12843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:46.787545 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 12726
I20260806 23:51:46.787721 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:51:46.788013 11142 external_mini_cluster.cc:1428] Setting key 82f7acca02ea3ecde25e53d2b738a53c
I20260806 23:51:46.788529 12843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc: Bootstrap starting.
I20260806 23:51:46.791891 12843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:46.793130 12843 log.cc:824] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:46.795881 12843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc: No bootstrap required, opened a new log
Aug 06 23:51:46 dist-test-slave-nj21 krb5kdc[12711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:46.801082 12843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a2989147ea4d4a8ac9af0e8eac42bc" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38125 } }
I20260806 23:51:46.801381 12843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:46.801453 12843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a2989147ea4d4a8ac9af0e8eac42bc, State: Initialized, Role: FOLLOWER
I20260806 23:51:46.801977 12843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a2989147ea4d4a8ac9af0e8eac42bc" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38125 } }
I20260806 23:51:46.802139 12843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:51:46.802222 12843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:51:46.802345 12843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:46.803562 12843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a2989147ea4d4a8ac9af0e8eac42bc" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38125 } }
I20260806 23:51:46.804008 12843 leader_election.cc:302] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14a2989147ea4d4a8ac9af0e8eac42bc; no voters:
I20260806 23:51:46.804467 12843 leader_election.cc:288] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:51:46.804569 12851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:46.804816 12851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [term 1 LEADER]: Becoming Leader. State: Replica: 14a2989147ea4d4a8ac9af0e8eac42bc, State: Running, Role: LEADER
I20260806 23:51:46.805253 12851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a2989147ea4d4a8ac9af0e8eac42bc" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38125 } }
I20260806 23:51:46.805914 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:46.789895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.10.225.190:38125)
0806 23:51:46.790404 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:46.790423 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:46.790468 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:51:46.791238 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:46.791258 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:46.791660 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:46.792066 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:46.793164 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:46.794253 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:46.795487 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:46.796113 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:46.796274 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:46.799922 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:46.799930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:46.799951 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:46.799998 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:46.801980 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:46.802610 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:46.802616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:46.802622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:46.802755 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:46.803050 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:46.803052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:46.803054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:46.803571 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:46.803820 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:46.804152 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 23:51:46.804644 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":149,"threads_started":1}
I20260806 23:51:46.806070 12843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:51:46.807130 12852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14a2989147ea4d4a8ac9af0e8eac42bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a2989147ea4d4a8ac9af0e8eac42bc" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38125 } } }
I20260806 23:51:46.807261 12852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:46.807541 12853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14a2989147ea4d4a8ac9af0e8eac42bc. Latest consensus state: current_term: 1 leader_uuid: "14a2989147ea4d4a8ac9af0e8eac42bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a2989147ea4d4a8ac9af0e8eac42bc" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38125 } } }
I20260806 23:51:46.807622 12853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:46.808599 12866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:51:46.811486 12866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:51:46.818710 12866 catalog_manager.cc:1347] Generated new cluster ID: de41e555588c44e896e0fc7759208818
I20260806 23:51:46.818831 12866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:51:46.831758 12866 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:51:46.833155 12866 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:51:46.844040 12866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc: Generated new TSK 0
I20260806 23:51:46.844995 12866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:51:46.879558 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38125
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:47.006924 12881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:47.007321 12881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:47.007398 12881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:47.012539 12881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:47.012645 12881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:47.012714 12881 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:51:47.012780 12881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:51:47.019871 12881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.225.190:38125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:47.021445 12881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:47.022821 12881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:47.034200 12886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:47.034204 12891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:47.034294 12887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:47.034548 12881 server_base.cc:1034] running on GCE node
I20260806 23:51:47.035001 12881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:47.035722 12881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:47.036927 12881 hybrid_clock.cc:648] HybridClock initialized: now 1786060307036910 us; error 44 us; skew 500 ppm
Aug 06 23:51:47 dist-test-slave-nj21 krb5kdc[12711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:47.040036 12881 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:51:47.040999 12881 webserver.cc:460] Webserver started at http://127.10.225.129:40303/ using document root <none> and password file <none>
I20260806 23:51:47.041570 12881 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:47.041635 12881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:47.041816 12881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:47.044068 12881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "e8707dbd557f403a8ba3e181269594cd"
format_stamp: "Formatted at 2026-08-06 23:51:47 on dist-test-slave-nj21"
server_key: "d6268e03a67e12981ea82726da25a34e"
server_key_iv: "ec0da8b97061b753164479de135e81fd"
server_key_version: "encryptionkey@0"
I20260806 23:51:47.044629 12881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "e8707dbd557f403a8ba3e181269594cd"
format_stamp: "Formatted at 2026-08-06 23:51:47 on dist-test-slave-nj21"
server_key: "d6268e03a67e12981ea82726da25a34e"
server_key_iv: "ec0da8b97061b753164479de135e81fd"
server_key_version: "encryptionkey@0"
I20260806 23:51:47.048810 12881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:51:47.051779 12902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:47.053155 12881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:51:47.053426 12881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "e8707dbd557f403a8ba3e181269594cd"
format_stamp: "Formatted at 2026-08-06 23:51:47 on dist-test-slave-nj21"
server_key: "d6268e03a67e12981ea82726da25a34e"
server_key_iv: "ec0da8b97061b753164479de135e81fd"
server_key_version: "encryptionkey@0"
I20260806 23:51:47.053694 12881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:47.083940 12881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:47.087338 12881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:47.087566 12881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:47.089205 12881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:47.089259 12881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:47.089316 12881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:47.089344 12881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:51:47 dist-test-slave-nj21 krb5kdc[12711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:47.099907 12881 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:39907
I20260806 23:51:47.099956 13069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:39907 every 8 connection(s)
I20260806 23:51:47.100970 12881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:51:47.104113 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.090062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:34887 (local address 127.10.225.190:38125)
0806 23:51:47.090215 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.090220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.091149 (+ 929us) server_negotiation.cc:407] Connection header received
0806 23:51:47.092265 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.092269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.092334 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.092392 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:47.093829 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.094420 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.095645 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.095951 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.096042 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.099565 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:47.099571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:47.099574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:47.099618 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:47.101560 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.102172 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.102176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.102177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.102267 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.102783 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.102786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.102788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.103016 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:47.103110 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.103832 (+ 722us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.103980 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:51:47.105144 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.090497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38125 (local address 127.10.225.129:34887)
0806 23:51:47.090973 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.091025 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.092011 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.092749 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.092755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.093099 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:47.093669 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.093683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.094546 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.094549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.095524 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.095535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.096123 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.096127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.096285 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.096899 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:47.096920 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:47.099274 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:47.101738 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.101745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.101761 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.102070 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.102421 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.102425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.102428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.102647 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.103159 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:47.103167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:47.103528 (+ 361us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.103778 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.104261 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260806 23:51:47.106606 13071 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38125
I20260806 23:51:47.106827 13071 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:47.107448 13071 heartbeater.cc:511] Master 127.10.225.190:38125 requested a full tablet report, sending...
I20260806 23:51:47.109054 12775 ts_manager.cc:194] Registered new tserver with Master: e8707dbd557f403a8ba3e181269594cd (127.10.225.129:39907)
I20260806 23:51:47.110183 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 12881
I20260806 23:51:47.110296 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:51:47.110566 12775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:34887
I20260806 23:51:47.110618 11142 external_mini_cluster.cc:1428] Setting key fc0ca4298c5438b234820d0cf00f8964
I20260806 23:51:47.116173 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.109157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:45407 (local address 127.10.225.190:38125)
0806 23:51:47.109335 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.109339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.109476 (+ 137us) server_negotiation.cc:407] Connection header received
0806 23:51:47.109591 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.109594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.109631 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.109692 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:47.110244 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.111018 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.112011 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.112220 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.112308 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.113413 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:47.113416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:47.113418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:47.113439 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:47.114674 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.115057 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.115060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.115061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.115132 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.115374 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.115377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.115379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.115542 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:47.115629 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.115910 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.116015 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260806 23:51:47.116185 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.109066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38125 (local address 127.10.225.129:45407)
0806 23:51:47.109374 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.109389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.109515 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.109724 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.109727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.109993 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:47.110125 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.110133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.111152 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.111155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.111912 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.111923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.112322 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.112326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.112425 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.112915 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:47.112927 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:47.113284 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:47.114777 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.114780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.114783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.114971 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.115213 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.115216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.115218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.115295 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.115633 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:47.115636 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:47.115750 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.115840 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.115984 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:51:47.140743 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38125
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:47.270814 13088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:47.271195 13088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:47.271272 13088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:47.275854 13088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:47.275931 13088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:47.275975 13088 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:51:47.276011 13088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:51:47.283195 13088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.225.190:38125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:47.284451 13088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:47.285564 13088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:51:47.296061 13088 server_base.cc:1034] running on GCE node
W20260806 23:51:47.296006 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:47.296006 13094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:47.296003 13105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:47.296648 13088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:47.297466 13088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:47.298748 13088 hybrid_clock.cc:648] HybridClock initialized: now 1786060307298724 us; error 61 us; skew 500 ppm
Aug 06 23:51:47 dist-test-slave-nj21 krb5kdc[12711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:47.301800 13088 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:51:47.302831 13088 webserver.cc:460] Webserver started at http://127.10.225.130:46241/ using document root <none> and password file <none>
I20260806 23:51:47.303436 13088 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:47.303517 13088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:47.303781 13088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:47.306186 13088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "db33850fd7d141bcb91a2013dfe8565d"
format_stamp: "Formatted at 2026-08-06 23:51:47 on dist-test-slave-nj21"
server_key: "25c6e41f1ed4cc54e2e00e876c85f131"
server_key_iv: "100434881e7617d41027bfa1c4bacc99"
server_key_version: "encryptionkey@0"
I20260806 23:51:47.306813 13088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "db33850fd7d141bcb91a2013dfe8565d"
format_stamp: "Formatted at 2026-08-06 23:51:47 on dist-test-slave-nj21"
server_key: "25c6e41f1ed4cc54e2e00e876c85f131"
server_key_iv: "100434881e7617d41027bfa1c4bacc99"
server_key_version: "encryptionkey@0"
I20260806 23:51:47.311352 13088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:51:47.314556 13116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:47.316428 13088 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 23:51:47.316668 13088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "db33850fd7d141bcb91a2013dfe8565d"
format_stamp: "Formatted at 2026-08-06 23:51:47 on dist-test-slave-nj21"
server_key: "25c6e41f1ed4cc54e2e00e876c85f131"
server_key_iv: "100434881e7617d41027bfa1c4bacc99"
server_key_version: "encryptionkey@0"
I20260806 23:51:47.316900 13088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:47.335300 13088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:47.339882 13088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:47.340134 13088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:47.342320 13088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:47.342394 13088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:47.342456 13088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:47.342478 13088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:51:47 dist-test-slave-nj21 krb5kdc[12711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:47.353319 13088 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:36619
I20260806 23:51:47.353340 13279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:36619 every 8 connection(s)
I20260806 23:51:47.354592 13088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:51:47.358860 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.343613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:55951 (local address 127.10.225.190:38125)
0806 23:51:47.343765 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.343770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.345080 (+ 1310us) server_negotiation.cc:407] Connection header received
0806 23:51:47.346613 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.346617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.346675 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.346775 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:47.348304 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.348954 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.350623 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.350847 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.350944 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.353929 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:47.353935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:47.353937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:47.353987 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:47.356052 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.356676 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.356680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.356681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.356774 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.357126 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.357129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.357130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.357533 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:47.357667 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.358513 (+ 846us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.358649 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 23:51:47.360105 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.344241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38125 (local address 127.10.225.130:55951)
0806 23:51:47.344894 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.344960 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.346384 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.346950 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.346961 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.347480 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:47.348151 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.348166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.349075 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.349079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.350470 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.350484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.351012 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.351016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.351219 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.351920 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:47.351941 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:47.353605 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:47.356241 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.356246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.356257 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.356532 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.356845 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.356847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.356849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.356997 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.357728 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:47.357737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:47.358116 (+ 379us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.358394 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.358838 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":145,"threads_started":1}
I20260806 23:51:47.361004 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 13088
I20260806 23:51:47.361150 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:51:47.361536 11142 external_mini_cluster.cc:1428] Setting key 0fecce3534fee67ec8ca24ad46afdb1b
I20260806 23:51:47.361871 13280 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38125
I20260806 23:51:47.362073 13280 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:47.362715 13280 heartbeater.cc:511] Master 127.10.225.190:38125 requested a full tablet report, sending...
I20260806 23:51:47.363960 12773 ts_manager.cc:194] Registered new tserver with Master: db33850fd7d141bcb91a2013dfe8565d (127.10.225.130:36619)
I20260806 23:51:47.364658 12773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:55951
I20260806 23:51:47.372390 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.364614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38125 (local address 127.10.225.130:49301)
0806 23:51:47.365214 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.365228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.365361 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.365704 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.365707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.365924 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:47.366027 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.366033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.366755 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.366759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.367317 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.367326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.367874 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.367878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.367955 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.368595 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:47.368607 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:47.368953 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:47.370789 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.370792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.370793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.370997 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.371305 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.371307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.371309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.371398 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.371927 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:47.371930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:47.372015 (+ 85us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.372101 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.372214 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":137,"threads_started":1}
I20260806 23:51:47.372462 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.364950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:49301 (local address 127.10.225.190:38125)
0806 23:51:47.365053 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.365057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.365290 (+ 233us) server_negotiation.cc:407] Connection header received
0806 23:51:47.365457 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.365460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.365504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.365551 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:47.366139 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.366639 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.367426 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.367725 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.367815 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.369102 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:47.369107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:47.369110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:47.369137 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:47.370672 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.371087 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.371091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.371093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.371184 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.371482 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.371484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.371486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.371826 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:47.371942 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.372178 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.372299 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:51:47.396763 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38125
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:47.522930 13290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:47.523223 13290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:47.523275 13290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:47.527447 13290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:47.527530 13290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:47.527571 13290 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:51:47.527606 13290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:51:47.533241 13290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.225.190:38125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:47.534435 13290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:47.535926 13290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:47.548503 13298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:47.548764 13290 server_base.cc:1034] running on GCE node
W20260806 23:51:47.548501 13299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:47.548501 13302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:47.549412 13290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:47.550202 13290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:47.551389 13290 hybrid_clock.cc:648] HybridClock initialized: now 1786060307551361 us; error 45 us; skew 500 ppm
Aug 06 23:51:47 dist-test-slave-nj21 krb5kdc[12711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:47.554687 13290 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:51:47.556195 13290 webserver.cc:460] Webserver started at http://127.10.225.131:36217/ using document root <none> and password file <none>
I20260806 23:51:47.556944 13290 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:47.557035 13290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:47.557302 13290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:47.560067 13290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "d4b5da63a68b498f878b6fd7eacad495"
format_stamp: "Formatted at 2026-08-06 23:51:47 on dist-test-slave-nj21"
server_key: "85e625cce115fa74f38842a1017f79c3"
server_key_iv: "6b3229d9da8428ef6c867d0ea0e3a698"
server_key_version: "encryptionkey@0"
I20260806 23:51:47.560719 13290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "d4b5da63a68b498f878b6fd7eacad495"
format_stamp: "Formatted at 2026-08-06 23:51:47 on dist-test-slave-nj21"
server_key: "85e625cce115fa74f38842a1017f79c3"
server_key_iv: "6b3229d9da8428ef6c867d0ea0e3a698"
server_key_version: "encryptionkey@0"
I20260806 23:51:47.565742 13290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 23:51:47.568773 13313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:47.570366 13290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:51:47.570509 13290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "d4b5da63a68b498f878b6fd7eacad495"
format_stamp: "Formatted at 2026-08-06 23:51:47 on dist-test-slave-nj21"
server_key: "85e625cce115fa74f38842a1017f79c3"
server_key_iv: "6b3229d9da8428ef6c867d0ea0e3a698"
server_key_version: "encryptionkey@0"
I20260806 23:51:47.570640 13290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:47.591063 13290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:47.594303 13290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:47.594502 13290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:47.596107 13290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:47.596159 13290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:47.596215 13290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:47.596240 13290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:51:47 dist-test-slave-nj21 krb5kdc[12711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060307, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:47.606516 13290 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:43627
I20260806 23:51:47.607782 13290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:51:47.607101 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:43627 every 8 connection(s)
I20260806 23:51:47.610114 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.596985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:54207 (local address 127.10.225.190:38125)
0806 23:51:47.597138 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.597143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.598026 (+ 883us) server_negotiation.cc:407] Connection header received
0806 23:51:47.599202 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.599206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.599271 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.599365 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:47.600714 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.601524 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.602859 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.603201 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.603321 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.606038 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:47.606042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:47.606045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:47.606073 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:47.607869 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.608509 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.608514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.608515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.608610 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.608948 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.608951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.608952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.609152 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:47.609240 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.609868 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.609982 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:51:47.610930 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.597391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38125 (local address 127.10.225.131:54207)
0806 23:51:47.597859 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.597911 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.598949 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.599560 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.599570 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.599989 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:47.600579 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.600595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.601657 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.601660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.602754 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.602765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.603376 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.603380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.603570 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.604183 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:47.604200 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:47.605834 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:47.608014 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.608020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.608035 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.608393 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.608710 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.608713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.608715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.608845 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.609280 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:47.609286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:47.609549 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.609729 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.610018 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260806 23:51:47.612653 13486 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38125
I20260806 23:51:47.612802 13486 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:47.613334 13486 heartbeater.cc:511] Master 127.10.225.190:38125 requested a full tablet report, sending...
I20260806 23:51:47.614434 12773 ts_manager.cc:194] Registered new tserver with Master: d4b5da63a68b498f878b6fd7eacad495 (127.10.225.131:43627)
I20260806 23:51:47.615087 12773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:54207
I20260806 23:51:47.617141 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 13290
I20260806 23:51:47.617297 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:51:47.617866 11142 external_mini_cluster.cc:1428] Setting key afcc0fe6cb3fd05ed9a2688b2b5553e9
I20260806 23:51:47.621068 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:51:47.622793 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.615364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:57011 (local address 127.10.225.190:38125)
0806 23:51:47.615509 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.615514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.615726 (+ 212us) server_negotiation.cc:407] Connection header received
0806 23:51:47.615882 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.615885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.615925 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.615982 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:47.616506 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.617003 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.617884 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.618133 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.618221 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.619604 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:47.619607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:47.619609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:47.619630 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:47.621085 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.621515 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.621519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.621521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.621598 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.621980 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.621983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.621985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.622177 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:47.622261 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.622520 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.622627 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 23:51:47.622804 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.615203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38125 (local address 127.10.225.131:57011)
0806 23:51:47.615612 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.615625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.615786 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.616018 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.616020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.616245 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:47.616400 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.616407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.617123 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.617126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.617773 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.617783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.618244 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.618248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.618340 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.618980 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:47.618992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:47.619471 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:47.621226 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.621230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.621232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.621426 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.621679 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.621682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.621684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.621868 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.622266 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:47.622269 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:47.622368 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.622456 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.622622 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":171,"threads_started":1}
Aug 06 23:51:47 dist-test-slave-nj21 krb5kdc[12711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:51:47 dist-test-slave-nj21 krb5kdc[12711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:47.652444 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.642712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.10.225.190:38125)
0806 23:51:47.642887 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.642892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.642908 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:51:47.642964 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.642966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.643017 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.643109 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:47.644007 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.644594 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.645462 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.645686 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.645787 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.648997 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:47.649002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:47.649004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:47.649045 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:47.650638 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.651131 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.651134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.651135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.651214 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.651517 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.651520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.651522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.651846 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:47.651957 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.652182 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.652299 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 23:51:47.655656 12773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:51:47.658914 12773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:51:47.675302 13521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.669938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.10.225.131:43627)
0806 23:51:47.670270 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.670282 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.670305 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:51:47.670378 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.670384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.670592 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.670749 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:47.671026 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.671890 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.673664 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.674450 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.674657 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.674790 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.675027 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.675150 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":123,"threads_started":1}
I20260806 23:51:47.675408 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.668841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:39907 (local address 127.0.0.1:58332)
0806 23:51:47.669716 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.669729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.669811 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.671014 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.671016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.671030 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:47.671097 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.671101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.672522 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.672526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.673756 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.673765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.674781 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.674785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.674896 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.674915 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.674964 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.675124 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":55,"thread_start_us":114,"threads_started":1}
I20260806 23:51:47.675660 13512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.668768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:43627 (local address 127.0.0.1:60546)
0806 23:51:47.669184 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.669231 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.669362 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.670750 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.670756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.670777 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:47.670916 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.670924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.671999 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.672003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.673518 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.673530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.674630 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.674634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.674861 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.674888 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.674959 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.675165 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":40,"thread_start_us":109,"threads_started":1}
I20260806 23:51:47.676440 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.670065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.10.225.129:39907)
0806 23:51:47.670341 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.670348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.670373 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:51:47.670449 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.670455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.670620 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.670748 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:47.671191 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.672393 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.673892 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.674632 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.676009 (+ 1377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.676140 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.676192 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.676298 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260806 23:51:47.680200 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.669700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.10.225.130:36619)
0806 23:51:47.670133 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.670149 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.670451 (+ 302us) server_negotiation.cc:407] Connection header received
0806 23:51:47.670546 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.670551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.670708 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.670826 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:47.671243 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.672398 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.673605 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.674215 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.674351 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.674513 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.679897 (+ 5384us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.680043 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":127,"threads_started":1}
I20260806 23:51:47.680732 13513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.669494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:36619 (local address 127.0.0.1:56144)
0806 23:51:47.670327 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.670350 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.670449 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.670950 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.670953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.670968 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:47.671085 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.671093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.672522 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.672526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.673480 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.673491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.674355 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.674359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.674521 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.674570 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.680481 (+ 5911us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.680587 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":122,"thread_start_us":93,"threads_started":1}
I20260806 23:51:47.680508 12984 tablet_service.cc:1467] Processing CreateTablet for tablet 64599708143147a8b5fd4a39b3b123b0 (DEFAULT_TABLE table=test-table [id=736eeec75d424fab9dd8143fc0670b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:47.681797 12984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64599708143147a8b5fd4a39b3b123b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:47.681660 13374 tablet_service.cc:1467] Processing CreateTablet for tablet 64599708143147a8b5fd4a39b3b123b0 (DEFAULT_TABLE table=test-table [id=736eeec75d424fab9dd8143fc0670b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:47.682718 13374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64599708143147a8b5fd4a39b3b123b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:47.686892 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 64599708143147a8b5fd4a39b3b123b0 (DEFAULT_TABLE table=test-table [id=736eeec75d424fab9dd8143fc0670b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:47.688098 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64599708143147a8b5fd4a39b3b123b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:47.689672 13530 tablet_bootstrap.cc:492] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: Bootstrap starting.
I20260806 23:51:47.692044 13531 tablet_bootstrap.cc:492] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: Bootstrap starting.
I20260806 23:51:47.692277 13530 tablet_bootstrap.cc:654] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:47.693262 13530 log.cc:824] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:47.694245 13531 tablet_bootstrap.cc:654] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:47.695078 13531 log.cc:824] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:47.695142 13530 tablet_bootstrap.cc:492] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: No bootstrap required, opened a new log
I20260806 23:51:47.695327 13530 ts_tablet_manager.cc:1397] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:51:47.696923 13531 tablet_bootstrap.cc:492] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: No bootstrap required, opened a new log
I20260806 23:51:47.697099 13531 ts_tablet_manager.cc:1397] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:51:47.698643 13530 raft_consensus.cc:348] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } }
I20260806 23:51:47.698931 13530 raft_consensus.cc:374] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:47.698993 13530 raft_consensus.cc:729] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b5da63a68b498f878b6fd7eacad495, State: Initialized, Role: FOLLOWER
I20260806 23:51:47.699389 13530 consensus_queue.cc:260] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } }
I20260806 23:51:47.699761 13534 tablet_bootstrap.cc:492] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: Bootstrap starting.
I20260806 23:51:47.700146 13486 heartbeater.cc:503] Master 127.10.225.190:38125 was elected leader, sending a full tablet report...
I20260806 23:51:47.700484 13530 ts_tablet_manager.cc:1428] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260806 23:51:47.701337 13531 raft_consensus.cc:348] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } }
I20260806 23:51:47.701622 13531 raft_consensus.cc:374] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:47.701680 13531 raft_consensus.cc:729] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8707dbd557f403a8ba3e181269594cd, State: Initialized, Role: FOLLOWER
I20260806 23:51:47.702122 13531 consensus_queue.cc:260] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } }
I20260806 23:51:47.702265 13534 tablet_bootstrap.cc:654] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:47.703179 13531 ts_tablet_manager.cc:1428] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:51:47.703178 13534 log.cc:824] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:47.703948 13071 heartbeater.cc:503] Master 127.10.225.190:38125 was elected leader, sending a full tablet report...
I20260806 23:51:47.705519 13534 tablet_bootstrap.cc:492] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: No bootstrap required, opened a new log
I20260806 23:51:47.705739 13534 ts_tablet_manager.cc:1397] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260806 23:51:47.709339 13534 raft_consensus.cc:348] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } }
I20260806 23:51:47.709612 13534 raft_consensus.cc:374] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:47.709648 13534 raft_consensus.cc:729] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db33850fd7d141bcb91a2013dfe8565d, State: Initialized, Role: FOLLOWER
I20260806 23:51:47.710042 13534 consensus_queue.cc:260] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } }
I20260806 23:51:47.710757 13280 heartbeater.cc:503] Master 127.10.225.190:38125 was elected leader, sending a full tablet report...
I20260806 23:51:47.710965 13534 ts_tablet_manager.cc:1428] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 23:51:47.737902 13538 raft_consensus.cc:482] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:51:47.738248 13538 raft_consensus.cc:504] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } }
I20260806 23:51:47.740016 13538 leader_election.cc:288] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4b5da63a68b498f878b6fd7eacad495 (127.10.225.131:43627), db33850fd7d141bcb91a2013dfe8565d (127.10.225.130:36619)
I20260806 23:51:47.746129 13521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.740487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:35885 (local address 127.10.225.131:43627)
0806 23:51:47.740658 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.740663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.741000 (+ 337us) server_negotiation.cc:407] Connection header received
0806 23:51:47.741183 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.741187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.741273 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.741336 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:47.741775 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.742671 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.744864 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.745503 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.745653 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.745715 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.745896 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.745988 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:51:47.746143 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.740369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:43627 (local address 127.10.225.129:35885)
0806 23:51:47.740858 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.740875 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.741039 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.741384 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.741388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.741441 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:47.741640 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.741648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.742917 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.742924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.744639 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.744661 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.745665 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.745669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.745767 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.745794 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.745863 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.745946 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":178,"spinlock_wait_cycles":5248,"thread_start_us":198,"threads_started":1}
I20260806 23:51:47.746528 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.740566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:49387 (local address 127.10.225.130:36619)
0806 23:51:47.740713 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.740718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.741066 (+ 348us) server_negotiation.cc:407] Connection header received
0806 23:51:47.741184 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.741187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.741251 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.741325 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:47.741775 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.742671 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.745295 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.745908 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.746037 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.746089 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.746296 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.746391 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 23:51:47.746912 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.740400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:36619 (local address 127.10.225.129:49387)
0806 23:51:47.740973 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.740984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.741069 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.741384 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.741388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.741439 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:47.741640 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.741648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.742917 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.742924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.744639 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.744661 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.746048 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.746052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.746140 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.746164 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.746698 (+ 534us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.746778 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":129,"spinlock_wait_cycles":44288,"thread_start_us":171,"threads_started":1}
I20260806 23:51:47.746937 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64599708143147a8b5fd4a39b3b123b0" candidate_uuid: "e8707dbd557f403a8ba3e181269594cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b5da63a68b498f878b6fd7eacad495" is_pre_election: true
I20260806 23:51:47.747296 13415 raft_consensus.cc:2454] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8707dbd557f403a8ba3e181269594cd in term 0.
I20260806 23:51:47.747499 13213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64599708143147a8b5fd4a39b3b123b0" candidate_uuid: "e8707dbd557f403a8ba3e181269594cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33850fd7d141bcb91a2013dfe8565d" is_pre_election: true
I20260806 23:51:47.747797 13213 raft_consensus.cc:2454] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8707dbd557f403a8ba3e181269594cd in term 0.
I20260806 23:51:47.747833 12908 leader_election.cc:302] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4b5da63a68b498f878b6fd7eacad495, e8707dbd557f403a8ba3e181269594cd; no voters:
I20260806 23:51:47.748143 13538 raft_consensus.cc:2781] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:51:47.748232 13538 raft_consensus.cc:482] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:51:47.748270 13538 raft_consensus.cc:3037] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:47.749439 13538 raft_consensus.cc:504] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } }
I20260806 23:51:47.749819 13538 leader_election.cc:288] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [CANDIDATE]: Term 1 election: Requested vote from peers d4b5da63a68b498f878b6fd7eacad495 (127.10.225.131:43627), db33850fd7d141bcb91a2013dfe8565d (127.10.225.130:36619)
I20260806 23:51:47.750216 13213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64599708143147a8b5fd4a39b3b123b0" candidate_uuid: "e8707dbd557f403a8ba3e181269594cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33850fd7d141bcb91a2013dfe8565d"
I20260806 23:51:47.750249 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64599708143147a8b5fd4a39b3b123b0" candidate_uuid: "e8707dbd557f403a8ba3e181269594cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b5da63a68b498f878b6fd7eacad495"
I20260806 23:51:47.750375 13213 raft_consensus.cc:3037] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:47.750404 13415 raft_consensus.cc:3037] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:47.751451 13415 raft_consensus.cc:2454] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8707dbd557f403a8ba3e181269594cd in term 1.
I20260806 23:51:47.751780 12908 leader_election.cc:302] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4b5da63a68b498f878b6fd7eacad495, e8707dbd557f403a8ba3e181269594cd; no voters:
I20260806 23:51:47.751940 13538 raft_consensus.cc:2781] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:47.751979 13213 raft_consensus.cc:2454] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8707dbd557f403a8ba3e181269594cd in term 1.
I20260806 23:51:47.752154 13538 raft_consensus.cc:686] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 1 LEADER]: Becoming Leader. State: Replica: e8707dbd557f403a8ba3e181269594cd, State: Running, Role: LEADER
I20260806 23:51:47.752542 13538 consensus_queue.cc:237] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } }
I20260806 23:51:47.755805 12773 catalog_manager.cc:5495] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd reported cstate change: term changed from 0 to 1, leader changed from <none> to e8707dbd557f403a8ba3e181269594cd (127.10.225.129). New cstate: current_term: 1 leader_uuid: "e8707dbd557f403a8ba3e181269594cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } health_report { overall_health: UNKNOWN } } }
I20260806 23:51:47.793144 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.789374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.10.225.129:39907)
0806 23:51:47.789580 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.789584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.789598 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:51:47.789676 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.789678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.789734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.789819 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:47.790117 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.790633 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.791531 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.791764 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.791854 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.792198 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:47.792713 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:47.792818 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.792927 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.792992 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 23:51:47.801669 13213 raft_consensus.cc:1261] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [term 1 FOLLOWER]: Refusing update from remote peer e8707dbd557f403a8ba3e181269594cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:47.802896 13546 consensus_queue.cc:1035] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:47.803643 13415 raft_consensus.cc:1261] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Refusing update from remote peer e8707dbd557f403a8ba3e181269594cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:47.804301 13546 consensus_queue.cc:1035] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:47.812103 13551 mvcc.cc:205] Tried to move back new op lower bound from 7315703020744085504 to 7315703020557258752. Current Snapshot: MvccSnapshot[applied={T|T < 7315703020744085504}]
I20260806 23:51:47.813687 13550 mvcc.cc:205] Tried to move back new op lower bound from 7315703020744085504 to 7315703020557258752. Current Snapshot: MvccSnapshot[applied={T|T < 7315703020744085504}]
I20260806 23:51:47.814286 13553 mvcc.cc:205] Tried to move back new op lower bound from 7315703020744085504 to 7315703020557258752. Current Snapshot: MvccSnapshot[applied={T|T < 7315703020744085504}]
I20260806 23:51:47.829119 12773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:47.829384 12773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:47.832295 12773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc: Sending DeleteTablet for 3 replicas of tablet 64599708143147a8b5fd4a39b3b123b0
I20260806 23:51:47.833357 13374 tablet_service.cc:1514] Processing DeleteTablet for tablet 64599708143147a8b5fd4a39b3b123b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:47 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:60546
I20260806 23:51:47.833357 12984 tablet_service.cc:1514] Processing DeleteTablet for tablet 64599708143147a8b5fd4a39b3b123b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:47 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:58332
I20260806 23:51:47.833962 13182 tablet_service.cc:1514] Processing DeleteTablet for tablet 64599708143147a8b5fd4a39b3b123b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:47 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:56144
I20260806 23:51:47.834342 13565 tablet_replica.cc:330] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: stopping tablet replica
I20260806 23:51:47.834386 12773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 23:51:47.834613 13565 raft_consensus.cc:2229] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 23:51:47.834828 12773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:51:47.834894 13565 raft_consensus.cc:2258] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:47.835186 13564 tablet_replica.cc:330] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: stopping tablet replica
I20260806 23:51:47.835410 13564 raft_consensus.cc:2229] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:47.835618 13564 raft_consensus.cc:2258] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:47.835919 13562 tablet_replica.cc:330] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: stopping tablet replica
I20260806 23:51:47.836120 13562 raft_consensus.cc:2229] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:51:47.836458 13562 raft_consensus.cc:2258] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:47.838147 13564 ts_tablet_manager.cc:1905] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:47.838148 13565 ts_tablet_manager.cc:1905] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:47.841323 13564 ts_tablet_manager.cc:1918] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:51:47.841451 13564 log.cc:1196] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/64599708143147a8b5fd4a39b3b123b0
I20260806 23:51:47.841770 13564 ts_tablet_manager.cc:1939] T 64599708143147a8b5fd4a39b3b123b0 P d4b5da63a68b498f878b6fd7eacad495: Deleting consensus metadata
I20260806 23:51:47.841900 13374 tablet_service.cc:1467] Processing CreateTablet for tablet 7d914de9bad44969a0cd7e263f4c76f5 (DEFAULT_TABLE table=test-table [id=f4f660d5185c4dc381c967df56baedac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:47.842289 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 7d914de9bad44969a0cd7e263f4c76f5 (DEFAULT_TABLE table=test-table [id=f4f660d5185c4dc381c967df56baedac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:47.842576 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d914de9bad44969a0cd7e263f4c76f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:47.842542 12984 tablet_service.cc:1467] Processing CreateTablet for tablet 7d914de9bad44969a0cd7e263f4c76f5 (DEFAULT_TABLE table=test-table [id=f4f660d5185c4dc381c967df56baedac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:47.842819 12984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d914de9bad44969a0cd7e263f4c76f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:47.843530 13374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d914de9bad44969a0cd7e263f4c76f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:47.845448 13531 tablet_bootstrap.cc:492] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: Bootstrap starting.
I20260806 23:51:47.845769 13530 tablet_bootstrap.cc:492] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: Bootstrap starting.
I20260806 23:51:47.846009 12755 catalog_manager.cc:4841] TS d4b5da63a68b498f878b6fd7eacad495 (127.10.225.131:43627): tablet 64599708143147a8b5fd4a39b3b123b0 (table test-table [id=736eeec75d424fab9dd8143fc0670b94]) successfully deleted
I20260806 23:51:47.846554 13531 tablet_bootstrap.cc:654] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:47.846724 13530 tablet_bootstrap.cc:654] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:47.848271 13531 tablet_bootstrap.cc:492] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: No bootstrap required, opened a new log
I20260806 23:51:47.848371 13531 ts_tablet_manager.cc:1397] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:51:47.848614 13562 ts_tablet_manager.cc:1905] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:47.848883 13531 raft_consensus.cc:348] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:47.849000 13531 raft_consensus.cc:374] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:47.849030 13531 raft_consensus.cc:729] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8707dbd557f403a8ba3e181269594cd, State: Initialized, Role: FOLLOWER
I20260806 23:51:47.849151 13531 consensus_queue.cc:260] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:47.849431 13531 ts_tablet_manager.cc:1428] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:47.849555 13534 tablet_bootstrap.cc:492] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: Bootstrap starting.
I20260806 23:51:47.850589 13534 tablet_bootstrap.cc:654] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: Neither blocks nor log segments found. Creating new log.
W20260806 23:51:47.852425 13073 tablet.cc:2369] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:51:47.852795 13562 ts_tablet_manager.cc:1918] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:51:47.852885 13562 log.cc:1196] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/64599708143147a8b5fd4a39b3b123b0
I20260806 23:51:47.853201 13562 ts_tablet_manager.cc:1939] T 64599708143147a8b5fd4a39b3b123b0 P e8707dbd557f403a8ba3e181269594cd: Deleting consensus metadata
I20260806 23:51:47.853396 13565 ts_tablet_manager.cc:1918] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:51:47.853492 13565 log.cc:1196] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/64599708143147a8b5fd4a39b3b123b0
I20260806 23:51:47.853790 13565 ts_tablet_manager.cc:1939] T 64599708143147a8b5fd4a39b3b123b0 P db33850fd7d141bcb91a2013dfe8565d: Deleting consensus metadata
I20260806 23:51:47.854770 12758 catalog_manager.cc:4841] TS e8707dbd557f403a8ba3e181269594cd (127.10.225.129:39907): tablet 64599708143147a8b5fd4a39b3b123b0 (table test-table [id=736eeec75d424fab9dd8143fc0670b94]) successfully deleted
I20260806 23:51:47.854959 12758 catalog_manager.cc:4841] TS db33850fd7d141bcb91a2013dfe8565d (127.10.225.130:36619): tablet 64599708143147a8b5fd4a39b3b123b0 (table test-table [id=736eeec75d424fab9dd8143fc0670b94]) successfully deleted
I20260806 23:51:47.855666 13534 tablet_bootstrap.cc:492] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: No bootstrap required, opened a new log
I20260806 23:51:47.855798 13534 ts_tablet_manager.cc:1397] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 23:51:47.856040 13530 tablet_bootstrap.cc:492] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: No bootstrap required, opened a new log
I20260806 23:51:47.856148 13530 ts_tablet_manager.cc:1397] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260806 23:51:47.856302 13534 raft_consensus.cc:348] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:47.856429 13534 raft_consensus.cc:374] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:47.856467 13534 raft_consensus.cc:729] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db33850fd7d141bcb91a2013dfe8565d, State: Initialized, Role: FOLLOWER
I20260806 23:51:47.856587 13534 consensus_queue.cc:260] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:47.856609 13530 raft_consensus.cc:348] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:47.856747 13530 raft_consensus.cc:374] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:47.856788 13530 raft_consensus.cc:729] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b5da63a68b498f878b6fd7eacad495, State: Initialized, Role: FOLLOWER
I20260806 23:51:47.856839 13534 ts_tablet_manager.cc:1428] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:47.856925 13530 consensus_queue.cc:260] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:47.857170 13530 ts_tablet_manager.cc:1428] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260806 23:51:47.858981 13489 tablet.cc:2369] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:51:47.921608 13540 raft_consensus.cc:482] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:51:47.921800 13540 raft_consensus.cc:504] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:47.923123 13540 leader_election.cc:288] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8707dbd557f403a8ba3e181269594cd (127.10.225.129:39907), d4b5da63a68b498f878b6fd7eacad495 (127.10.225.131:43627)
I20260806 23:51:47.927896 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.923380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:43627 (local address 127.10.225.130:42733)
0806 23:51:47.923913 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.923925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.924041 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.924330 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.924333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.924352 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:47.924537 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.924543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.925541 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.925544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.926617 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.926624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.927462 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.927465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.927556 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.927576 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.927641 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.927736 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":236,"spinlock_wait_cycles":40320,"thread_start_us":93,"threads_started":1}
I20260806 23:51:47.927877 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.923498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:36513 (local address 127.10.225.129:39907)
0806 23:51:47.923647 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.923651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.923919 (+ 268us) server_negotiation.cc:407] Connection header received
0806 23:51:47.924167 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.924170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.924214 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.924249 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:47.924661 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.925171 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.926715 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.927156 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.927235 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.927284 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.927612 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.927725 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:51:47.927897 13521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.923775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:42733 (local address 127.10.225.131:43627)
0806 23:51:47.923903 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.923907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.924000 (+ 93us) server_negotiation.cc:407] Connection header received
0806 23:51:47.924153 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.924157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.924213 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.924304 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:47.924638 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.925412 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.926741 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.927341 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.927466 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.927515 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.927665 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.927765 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:51:47.928261 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.923380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:39907 (local address 127.10.225.130:36513)
0806 23:51:47.923819 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.923837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.924041 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.924381 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.924383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.924396 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:47.924532 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.924539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.925294 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.925298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.926560 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.926573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.927367 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.927370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.927470 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.927498 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.928046 (+ 548us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.928126 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":10,"spinlock_wait_cycles":32512,"thread_start_us":88,"threads_started":1}
I20260806 23:51:47.928611 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d914de9bad44969a0cd7e263f4c76f5" candidate_uuid: "db33850fd7d141bcb91a2013dfe8565d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b5da63a68b498f878b6fd7eacad495" is_pre_election: true
I20260806 23:51:47.928785 13415 raft_consensus.cc:2454] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db33850fd7d141bcb91a2013dfe8565d in term 0.
I20260806 23:51:47.928864 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d914de9bad44969a0cd7e263f4c76f5" candidate_uuid: "db33850fd7d141bcb91a2013dfe8565d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8707dbd557f403a8ba3e181269594cd" is_pre_election: true
I20260806 23:51:47.929106 13013 raft_consensus.cc:2454] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db33850fd7d141bcb91a2013dfe8565d in term 0.
I20260806 23:51:47.929224 13118 leader_election.cc:302] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4b5da63a68b498f878b6fd7eacad495, db33850fd7d141bcb91a2013dfe8565d; no voters:
I20260806 23:51:47.929562 13540 raft_consensus.cc:2781] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:51:47.929622 13540 raft_consensus.cc:482] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:51:47.929664 13540 raft_consensus.cc:3037] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:47.930666 13540 raft_consensus.cc:504] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:47.931038 13540 leader_election.cc:288] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [CANDIDATE]: Term 1 election: Requested vote from peers e8707dbd557f403a8ba3e181269594cd (127.10.225.129:39907), d4b5da63a68b498f878b6fd7eacad495 (127.10.225.131:43627)
I20260806 23:51:47.931380 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d914de9bad44969a0cd7e263f4c76f5" candidate_uuid: "db33850fd7d141bcb91a2013dfe8565d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b5da63a68b498f878b6fd7eacad495"
I20260806 23:51:47.931401 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d914de9bad44969a0cd7e263f4c76f5" candidate_uuid: "db33850fd7d141bcb91a2013dfe8565d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8707dbd557f403a8ba3e181269594cd"
I20260806 23:51:47.931505 13415 raft_consensus.cc:3037] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:47.931546 13013 raft_consensus.cc:3037] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:47.932493 13415 raft_consensus.cc:2454] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db33850fd7d141bcb91a2013dfe8565d in term 1.
I20260806 23:51:47.932758 13118 leader_election.cc:302] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4b5da63a68b498f878b6fd7eacad495, db33850fd7d141bcb91a2013dfe8565d; no voters:
I20260806 23:51:47.932818 13013 raft_consensus.cc:2454] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db33850fd7d141bcb91a2013dfe8565d in term 1.
I20260806 23:51:47.932948 13540 raft_consensus.cc:2781] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:47.933154 13540 raft_consensus.cc:686] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 1 LEADER]: Becoming Leader. State: Replica: db33850fd7d141bcb91a2013dfe8565d, State: Running, Role: LEADER
I20260806 23:51:47.933435 13540 consensus_queue.cc:237] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:47.936065 12773 catalog_manager.cc:5495] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d reported cstate change: term changed from 0 to 1, leader changed from <none> to db33850fd7d141bcb91a2013dfe8565d (127.10.225.130). New cstate: current_term: 1 leader_uuid: "db33850fd7d141bcb91a2013dfe8565d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } health_report { overall_health: UNKNOWN } } }
Aug 06 23:51:47 dist-test-slave-nj21 krb5kdc[12711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060306, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:47.994591 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.985624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38125 (local address 127.0.0.1:37478)
0806 23:51:47.985994 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:47.986017 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:47.986155 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:47.986405 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:47.986408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:47.986913 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:47.987063 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.987071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.987938 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.987942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.988683 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:47.988693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.989147 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.989150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:47.989227 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.989870 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:47.989884 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:47.991544 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:47.993089 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.993098 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.993104 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.993362 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.993637 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:47.993640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:47.993641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:47.993728 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:47.994082 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:47.994088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:47.994199 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 23:51:47.994298 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 23:51:47.994438 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":110,"threads_started":1}
I20260806 23:51:47.994953 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:47.985769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.10.225.190:38125)
0806 23:51:47.985905 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:47.985910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:47.986102 (+ 192us) server_negotiation.cc:407] Connection header received
0806 23:51:47.986251 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:47.986253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:47.986298 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:47.986359 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:47.987211 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.987804 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.988820 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:47.989042 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:47.989130 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:47.991729 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:47.991732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:47.991734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:47.991757 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:47.992975 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.993455 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.993458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.993460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.993536 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:47.993795 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:47.993797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:47.993798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:47.993986 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:47.994054 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:47.994537 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 23:51:47.994746 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 23:51:48.000473 12773 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:37478:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 23:51:48.001554 12773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:51:48.007618 12984 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3a41d1305c4c36861d6c9b3a52a516 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4a84dca8567c43c09b9e2a3f5c36b4ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:51:48.007943 13374 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3a41d1305c4c36861d6c9b3a52a516 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4a84dca8567c43c09b9e2a3f5c36b4ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:51:48.008260 12984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3a41d1305c4c36861d6c9b3a52a516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:48.008298 13374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3a41d1305c4c36861d6c9b3a52a516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:48.007660 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3a41d1305c4c36861d6c9b3a52a516 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4a84dca8567c43c09b9e2a3f5c36b4ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:51:48.010977 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3a41d1305c4c36861d6c9b3a52a516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:48.011752 13530 tablet_bootstrap.cc:492] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: Bootstrap starting.
I20260806 23:51:48.012557 13531 tablet_bootstrap.cc:492] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: Bootstrap starting.
I20260806 23:51:48.012950 13530 tablet_bootstrap.cc:654] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:48.013396 13531 tablet_bootstrap.cc:654] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:48.014245 13530 tablet_bootstrap.cc:492] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: No bootstrap required, opened a new log
I20260806 23:51:48.014335 13530 ts_tablet_manager.cc:1397] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:51:48.014659 13534 tablet_bootstrap.cc:492] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: Bootstrap starting.
I20260806 23:51:48.014796 13530 raft_consensus.cc:348] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:48.014931 13530 raft_consensus.cc:374] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:48.014953 13530 raft_consensus.cc:729] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b5da63a68b498f878b6fd7eacad495, State: Initialized, Role: FOLLOWER
I20260806 23:51:48.015079 13530 consensus_queue.cc:260] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:48.015334 13530 ts_tablet_manager.cc:1428] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:48.015427 13537 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.015558 13537 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:48.015694 13534 tablet_bootstrap.cc:654] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:48.016955 13531 tablet_bootstrap.cc:492] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: No bootstrap required, opened a new log
I20260806 23:51:48.017056 13531 ts_tablet_manager.cc:1397] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260806 23:51:48.017400 13534 tablet_bootstrap.cc:492] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: No bootstrap required, opened a new log
I20260806 23:51:48.017500 13534 ts_tablet_manager.cc:1397] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:51:48.017455 13531 raft_consensus.cc:348] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:48.017573 13531 raft_consensus.cc:374] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:48.017606 13531 raft_consensus.cc:729] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8707dbd557f403a8ba3e181269594cd, State: Initialized, Role: FOLLOWER
I20260806 23:51:48.017733 13531 consensus_queue.cc:260] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:48.017931 13534 raft_consensus.cc:348] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:48.018038 13534 raft_consensus.cc:374] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:48.018041 13531 ts_tablet_manager.cc:1428] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:48.018069 13534 raft_consensus.cc:729] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db33850fd7d141bcb91a2013dfe8565d, State: Initialized, Role: FOLLOWER
I20260806 23:51:48.018186 13534 consensus_queue.cc:260] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:48.018024 13546 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.018353 13546 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:48.018538 13534 ts_tablet_manager.cc:1428] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:51:48.018488 13540 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.018615 13540 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:48.063946 13540 raft_consensus.cc:482] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:51:48.064106 13540 raft_consensus.cc:504] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:48.064553 13540 leader_election.cc:288] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8707dbd557f403a8ba3e181269594cd (127.10.225.129:39907), d4b5da63a68b498f878b6fd7eacad495 (127.10.225.131:43627)
I20260806 23:51:48.065079 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3a41d1305c4c36861d6c9b3a52a516" candidate_uuid: "db33850fd7d141bcb91a2013dfe8565d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8707dbd557f403a8ba3e181269594cd" is_pre_election: true
I20260806 23:51:48.065148 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3a41d1305c4c36861d6c9b3a52a516" candidate_uuid: "db33850fd7d141bcb91a2013dfe8565d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b5da63a68b498f878b6fd7eacad495" is_pre_election: true
I20260806 23:51:48.065234 13013 raft_consensus.cc:2454] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db33850fd7d141bcb91a2013dfe8565d in term 0.
I20260806 23:51:48.065330 13415 raft_consensus.cc:2454] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db33850fd7d141bcb91a2013dfe8565d in term 0.
I20260806 23:51:48.065580 13120 leader_election.cc:302] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db33850fd7d141bcb91a2013dfe8565d, e8707dbd557f403a8ba3e181269594cd; no voters:
I20260806 23:51:48.065754 13540 raft_consensus.cc:2781] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:51:48.065815 13540 raft_consensus.cc:482] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:51:48.065850 13540 raft_consensus.cc:3037] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:48.066862 13540 raft_consensus.cc:504] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:48.067174 13540 leader_election.cc:288] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [CANDIDATE]: Term 1 election: Requested vote from peers e8707dbd557f403a8ba3e181269594cd (127.10.225.129:39907), d4b5da63a68b498f878b6fd7eacad495 (127.10.225.131:43627)
I20260806 23:51:48.067529 13415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3a41d1305c4c36861d6c9b3a52a516" candidate_uuid: "db33850fd7d141bcb91a2013dfe8565d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b5da63a68b498f878b6fd7eacad495"
I20260806 23:51:48.067531 13013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3a41d1305c4c36861d6c9b3a52a516" candidate_uuid: "db33850fd7d141bcb91a2013dfe8565d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8707dbd557f403a8ba3e181269594cd"
I20260806 23:51:48.067648 13013 raft_consensus.cc:3037] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:48.067651 13415 raft_consensus.cc:3037] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:48.068625 13415 raft_consensus.cc:2454] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db33850fd7d141bcb91a2013dfe8565d in term 1.
I20260806 23:51:48.068746 13013 raft_consensus.cc:2454] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db33850fd7d141bcb91a2013dfe8565d in term 1.
I20260806 23:51:48.068890 13118 leader_election.cc:302] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4b5da63a68b498f878b6fd7eacad495, db33850fd7d141bcb91a2013dfe8565d; no voters:
I20260806 23:51:48.069065 13540 raft_consensus.cc:2781] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:48.069154 13540 raft_consensus.cc:686] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [term 1 LEADER]: Becoming Leader. State: Replica: db33850fd7d141bcb91a2013dfe8565d, State: Running, Role: LEADER
I20260806 23:51:48.069290 13540 consensus_queue.cc:237] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } }
I20260806 23:51:48.070218 13573 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: TxnStatusTablet state changed. Reason: New leader db33850fd7d141bcb91a2013dfe8565d. Latest consensus state: current_term: 1 leader_uuid: "db33850fd7d141bcb91a2013dfe8565d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.070344 13573 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:51:48.070765 13585 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 23:51:48.071017 13585 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 23:51:48.071887 12775 catalog_manager.cc:5495] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d reported cstate change: term changed from 0 to 1, leader changed from <none> to db33850fd7d141bcb91a2013dfe8565d (127.10.225.130). New cstate: current_term: 1 leader_uuid: "db33850fd7d141bcb91a2013dfe8565d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } health_report { overall_health: UNKNOWN } } }
I20260806 23:51:48.087230 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:48.082390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:36619 (local address 127.0.0.1:56160)
0806 23:51:48.082599 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:48.082615 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:48.082718 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:48.083207 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:48.083210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:48.083223 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:51:48.083392 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:48.083398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.084512 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.084516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:48.085164 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:48.085173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.085801 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.085806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:48.085909 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:48.085988 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:51:48.086889 (+ 901us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:51:48.086920 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 23:51:48.086993 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 23:51:48.087105 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 23:51:48.087232 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:48.082534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.10.225.130:36619)
0806 23:51:48.082683 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:48.082688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:48.082706 (+ 18us) server_negotiation.cc:407] Connection header received
0806 23:51:48.082848 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:48.082851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:48.082921 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:48.083027 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:48.083531 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.084355 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.085267 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.085618 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.085723 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:48.086162 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:48.086753 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:48.086877 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:48.087017 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 23:51:48.087081 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:51:48.090477 13415 raft_consensus.cc:1261] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Refusing update from remote peer db33850fd7d141bcb91a2013dfe8565d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:48.090610 13013 raft_consensus.cc:1261] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd [term 1 FOLLOWER]: Refusing update from remote peer db33850fd7d141bcb91a2013dfe8565d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:48.091009 13573 consensus_queue.cc:1035] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:48.091174 13540 consensus_queue.cc:1035] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:48.092746 13537 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: TxnStatusTablet state changed. Reason: New leader db33850fd7d141bcb91a2013dfe8565d. Latest consensus state: current_term: 1 leader_uuid: "db33850fd7d141bcb91a2013dfe8565d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.092883 13537 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:48.092927 13546 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: TxnStatusTablet state changed. Reason: New leader db33850fd7d141bcb91a2013dfe8565d. Latest consensus state: current_term: 1 leader_uuid: "db33850fd7d141bcb91a2013dfe8565d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.093039 13546 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:48.096081 13546 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db33850fd7d141bcb91a2013dfe8565d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.096200 13546 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P e8707dbd557f403a8ba3e181269594cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:48.096221 13540 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db33850fd7d141bcb91a2013dfe8565d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.096323 13540 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:51:48.096684 13537 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db33850fd7d141bcb91a2013dfe8565d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.096805 13537 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P d4b5da63a68b498f878b6fd7eacad495: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:48.097046 13540 tablet_replica.cc:439] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db33850fd7d141bcb91a2013dfe8565d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 } } peers { permanent_uuid: "db33850fd7d141bcb91a2013dfe8565d" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 36619 } } peers { permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 } } }
I20260806 23:51:48.097146 13540 tablet_replica.cc:442] T 6f3a41d1305c4c36861d6c9b3a52a516 P db33850fd7d141bcb91a2013dfe8565d: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:51:48.105808 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:48.102637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.10.225.130:36619)
0806 23:51:48.102738 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:48.102742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:48.102798 (+ 56us) server_negotiation.cc:407] Connection header received
0806 23:51:48.102942 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:48.102945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:48.102991 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:48.103054 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:48.103411 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.104040 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.104830 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.105063 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.105148 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:48.105383 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:48.105459 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:48.105514 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:48.105606 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 23:51:48.105659 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 23:51:48.106559 13281 tablet.cc:2369] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:51:48.124984 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:48.113508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:37621 (local address 127.10.225.130:36619)
0806 23:51:48.113613 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:48.113618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:48.120385 (+ 6767us) server_negotiation.cc:407] Connection header received
0806 23:51:48.120568 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:48.120571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:48.120620 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:48.120687 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:48.120992 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.121546 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.123434 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.123905 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.124026 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:48.124453 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:48.124551 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:48.124614 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:48.124708 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 23:51:48.124805 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":36}
I20260806 23:51:48.124997 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:48.113464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:36619 (local address 127.10.225.130:37621)
0806 23:51:48.120194 (+ 6730us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:48.120225 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:48.120446 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:48.120714 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:48.120716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:48.120725 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:51:48.120898 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:48.120904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.121693 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.121696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:48.123295 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:48.123308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.124160 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.124162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:48.124242 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:48.124297 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:51:48.124629 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:51:48.124635 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:51:48.124694 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 23:51:48.124845 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6402,"mutex_wait_us":50,"thread_start_us":110,"threads_started":1}
I20260806 23:51:48.134456 13415 raft_consensus.cc:1261] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Refusing update from remote peer db33850fd7d141bcb91a2013dfe8565d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:48.134532 13013 raft_consensus.cc:1261] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [term 1 FOLLOWER]: Refusing update from remote peer db33850fd7d141bcb91a2013dfe8565d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:48.135157 13540 consensus_queue.cc:1035] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8707dbd557f403a8ba3e181269594cd" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:48.135380 13573 consensus_queue.cc:1035] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b5da63a68b498f878b6fd7eacad495" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 43627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:48.184293 12775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:48.184486 12775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:48.186743 12775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14a2989147ea4d4a8ac9af0e8eac42bc: Sending DeleteTablet for 3 replicas of tablet 7d914de9bad44969a0cd7e263f4c76f5
I20260806 23:51:48.187450 13181 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d914de9bad44969a0cd7e263f4c76f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:48 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:56144
I20260806 23:51:48.187458 12984 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d914de9bad44969a0cd7e263f4c76f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:48 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:58332
I20260806 23:51:48.187460 13374 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d914de9bad44969a0cd7e263f4c76f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:48 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:60546
I20260806 23:51:48.187657 13562 tablet_replica.cc:330] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: stopping tablet replica
I20260806 23:51:48.187661 13565 tablet_replica.cc:330] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: stopping tablet replica
I20260806 23:51:48.187722 13564 tablet_replica.cc:330] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: stopping tablet replica
I20260806 23:51:48.187810 13565 raft_consensus.cc:2229] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:51:48.187817 13562 raft_consensus.cc:2229] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:48.187839 13564 raft_consensus.cc:2229] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:48.187954 13564 raft_consensus.cc:2258] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:48.187963 13562 raft_consensus.cc:2258] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:48.188143 13565 raft_consensus.cc:2258] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:48.188841 13562 ts_tablet_manager.cc:1905] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:48.188927 13565 ts_tablet_manager.cc:1905] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:48.188999 13564 ts_tablet_manager.cc:1905] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:48.191890 13565 ts_tablet_manager.cc:1918] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:51:48.191890 13562 ts_tablet_manager.cc:1918] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:51:48.191912 13564 ts_tablet_manager.cc:1918] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:51:48.191967 13562 log.cc:1196] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/7d914de9bad44969a0cd7e263f4c76f5
I20260806 23:51:48.191967 13565 log.cc:1196] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/7d914de9bad44969a0cd7e263f4c76f5
I20260806 23:51:48.191988 13564 log.cc:1196] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/7d914de9bad44969a0cd7e263f4c76f5
I20260806 23:51:48.192227 13565 ts_tablet_manager.cc:1939] T 7d914de9bad44969a0cd7e263f4c76f5 P db33850fd7d141bcb91a2013dfe8565d: Deleting consensus metadata
I20260806 23:51:48.192234 13562 ts_tablet_manager.cc:1939] T 7d914de9bad44969a0cd7e263f4c76f5 P e8707dbd557f403a8ba3e181269594cd: Deleting consensus metadata
I20260806 23:51:48.192236 13564 ts_tablet_manager.cc:1939] T 7d914de9bad44969a0cd7e263f4c76f5 P d4b5da63a68b498f878b6fd7eacad495: Deleting consensus metadata
I20260806 23:51:48.193125 12755 catalog_manager.cc:4841] TS d4b5da63a68b498f878b6fd7eacad495 (127.10.225.131:43627): tablet 7d914de9bad44969a0cd7e263f4c76f5 (table test-table [id=f4f660d5185c4dc381c967df56baedac]) successfully deleted
I20260806 23:51:48.193125 12758 catalog_manager.cc:4841] TS e8707dbd557f403a8ba3e181269594cd (127.10.225.129:39907): tablet 7d914de9bad44969a0cd7e263f4c76f5 (table test-table [id=f4f660d5185c4dc381c967df56baedac]) successfully deleted
I20260806 23:51:48.193759 12758 catalog_manager.cc:4841] TS db33850fd7d141bcb91a2013dfe8565d (127.10.225.130:36619): tablet 7d914de9bad44969a0cd7e263f4c76f5 (table test-table [id=f4f660d5185c4dc381c967df56baedac]) successfully deleted
Aug 06 23:51:48 dist-test-slave-nj21 krb5kdc[12711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:51:48.206487 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:48.194306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.10.225.129:39907)
0806 23:51:48.194420 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:48.194425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:48.194651 (+ 226us) server_negotiation.cc:407] Connection header received
0806 23:51:48.194784 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:48.194787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:48.194834 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:48.194902 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:48.195579 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.196156 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.196999 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.197225 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.197323 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:48.201788 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:48.201802 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:48.201820 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:48.201853 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:48.204030 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:48.204646 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:48.204652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:48.204654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:48.204757 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:48.205050 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:48.205055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:48.205056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:48.205400 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:48.205910 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:48.206141 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 23:51:48.206301 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 23:51:48.207197 13061 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58344
I20260806 23:51:48.219272 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:48.209600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.10.225.190:38125)
0806 23:51:48.209756 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:48.209760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:48.209777 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:51:48.209878 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:48.209881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:48.209942 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:48.210029 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:48.210756 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.211529 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.212479 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:48.212810 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:48.212937 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:48.214851 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:48.214856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:48.214858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:48.214893 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:48.216836 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:48.217377 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:48.217381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:48.217383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:48.217498 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:48.217984 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:48.217990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:48.217992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:48.218393 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:48.218586 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:48.218882 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 23:51:48.219058 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 23:51:48.219910 12775 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37484
I20260806 23:51:48.222043 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 12881
I20260806 23:51:48.240888 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 13088
I20260806 23:51:48.255255 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 13290
I20260806 23:51:48.270419 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 12726
2026-08-06T23:51:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3025 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:51:48 dist-test-slave-nj21 krb5kdc[13608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:51:48 dist-test-slave-nj21 krb5kdc[13608](info): set up 2 sockets
Aug 06 23:51:48 dist-test-slave-nj21 krb5kdc[13608](info): commencing operation
krb5kdc: starting...
W20260806 23:51:49.442502 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:51:49 dist-test-slave-nj21 krb5kdc[13608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:51:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:51:49.588335 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38301
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:38301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:49.720752 13630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:49.721161 13630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:49.721230 13630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:49.726046 13630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:51:49.726171 13630 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 23:51:49.726202 13630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:49.726229 13630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:49.726251 13630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:51:49.726277 13630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:51:49.733515 13630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:38301
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:49.734798 13630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:49.735996 13630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:49.745747 13642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:49.746145 13630 server_base.cc:1034] running on GCE node
W20260806 23:51:49.745764 13641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:49.745764 13646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:49.747002 13630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:49.748394 13630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:49.749596 13630 hybrid_clock.cc:648] HybridClock initialized: now 1786060309749584 us; error 35 us; skew 500 ppm
Aug 06 23:51:49 dist-test-slave-nj21 krb5kdc[13608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060309, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:49.752821 13630 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:51:49.753844 13630 webserver.cc:460] Webserver started at http://127.10.225.190:41275/ using document root <none> and password file <none>
I20260806 23:51:49.754456 13630 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:49.754546 13630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:49.754794 13630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:49.757078 13630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "b90a5f761c954125a3b67a716e8ac014"
format_stamp: "Formatted at 2026-08-06 23:51:49 on dist-test-slave-nj21"
server_key: "4ebcb27824d61367a0cffe06032fc809"
server_key_iv: "aeca2382fc2af738e9909ab68e533cdc"
server_key_version: "encryptionkey@0"
I20260806 23:51:49.757690 13630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "b90a5f761c954125a3b67a716e8ac014"
format_stamp: "Formatted at 2026-08-06 23:51:49 on dist-test-slave-nj21"
server_key: "4ebcb27824d61367a0cffe06032fc809"
server_key_iv: "aeca2382fc2af738e9909ab68e533cdc"
server_key_version: "encryptionkey@0"
I20260806 23:51:49.762179 13630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:51:49.765123 13654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:49.766534 13630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:51:49.766677 13630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "b90a5f761c954125a3b67a716e8ac014"
format_stamp: "Formatted at 2026-08-06 23:51:49 on dist-test-slave-nj21"
server_key: "4ebcb27824d61367a0cffe06032fc809"
server_key_iv: "aeca2382fc2af738e9909ab68e533cdc"
server_key_version: "encryptionkey@0"
I20260806 23:51:49.766799 13630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:49.795612 13630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:49.799940 13630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:49.800145 13630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:49.808998 13630 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:38301
I20260806 23:51:49.809028 13754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:38301 every 8 connection(s)
I20260806 23:51:49.810050 13630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:51:49.813189 13755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:49.817303 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 13630
I20260806 23:51:49.817422 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:51:49.817696 11142 external_mini_cluster.cc:1428] Setting key 649698520efc394d8ae5d42c2905e223
I20260806 23:51:49.819612 13755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: Bootstrap starting.
I20260806 23:51:49.822057 13755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:49.822846 13755 log.cc:824] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:49.824880 13755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: No bootstrap required, opened a new log
Aug 06 23:51:49 dist-test-slave-nj21 krb5kdc[13608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:49.828651 13755 raft_consensus.cc:348] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a5f761c954125a3b67a716e8ac014" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38301 } }
I20260806 23:51:49.828915 13755 raft_consensus.cc:374] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:49.828953 13755 raft_consensus.cc:729] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90a5f761c954125a3b67a716e8ac014, State: Initialized, Role: FOLLOWER
I20260806 23:51:49.829407 13755 consensus_queue.cc:260] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a5f761c954125a3b67a716e8ac014" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38301 } }
I20260806 23:51:49.829555 13755 raft_consensus.cc:388] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:51:49.829757 13755 raft_consensus.cc:482] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:51:49.829967 13755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:49.831136 13755 raft_consensus.cc:504] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a5f761c954125a3b67a716e8ac014" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38301 } }
I20260806 23:51:49.831492 13755 leader_election.cc:302] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90a5f761c954125a3b67a716e8ac014; no voters:
I20260806 23:51:49.831823 13755 leader_election.cc:288] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:51:49.831939 13764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:49.832167 13764 raft_consensus.cc:686] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [term 1 LEADER]: Becoming Leader. State: Replica: b90a5f761c954125a3b67a716e8ac014, State: Running, Role: LEADER
I20260806 23:51:49.832500 13764 consensus_queue.cc:237] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a5f761c954125a3b67a716e8ac014" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38301 } }
I20260806 23:51:49.833258 13755 sys_catalog.cc:564] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:51:49.834759 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:49.819484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.10.225.190:38301)
0806 23:51:49.820109 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:49.820132 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:49.820188 (+ 56us) server_negotiation.cc:407] Connection header received
0806 23:51:49.821193 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:49.821225 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:49.821754 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:49.822203 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:49.823264 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:49.824476 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:49.825491 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:49.826200 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:49.826421 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:49.829394 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:49.829399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:49.829416 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:49.829448 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:49.831148 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:49.831604 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:49.831611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:49.831618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:49.831797 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:49.832214 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:49.832217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:49.832219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:49.832765 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:49.832934 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:49.833405 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 23:51:49.833731 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":177,"threads_started":1}
I20260806 23:51:49.835850 13767 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b90a5f761c954125a3b67a716e8ac014. Latest consensus state: current_term: 1 leader_uuid: "b90a5f761c954125a3b67a716e8ac014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a5f761c954125a3b67a716e8ac014" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38301 } } }
I20260806 23:51:49.837281 13767 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:49.835877 13765 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b90a5f761c954125a3b67a716e8ac014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90a5f761c954125a3b67a716e8ac014" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38301 } } }
I20260806 23:51:49.837844 13765 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014 [sys.catalog]: This master's current role is: LEADER
W20260806 23:51:49.840123 13787 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:51:49.840193 13787 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:51:49.840266 13789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:51:49.841045 13789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:51:49.847916 13789 catalog_manager.cc:1347] Generated new cluster ID: d848221307a942a4947833a48b4aefdc
I20260806 23:51:49.848014 13789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:51:49.874821 13789 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:51:49.876319 13789 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:51:49.883050 13789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: Generated new TSK 0
I20260806 23:51:49.883983 13789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:51:49.919723 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38301
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:50.049716 13796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:50.050037 13796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:50.050091 13796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:50.054265 13796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:50.054345 13796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:50.054387 13796 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 23:51:50.054401 13796 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:51:50.054435 13796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:51:50.061172 13796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.225.190:38301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:50.062764 13796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:50.064162 13796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:50.077065 13804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:50.077111 13808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:50.077293 13796 server_base.cc:1034] running on GCE node
W20260806 23:51:50.077065 13805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:50.078120 13796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:50.079121 13796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:50.080354 13796 hybrid_clock.cc:648] HybridClock initialized: now 1786060310080333 us; error 33 us; skew 500 ppm
Aug 06 23:51:50 dist-test-slave-nj21 krb5kdc[13608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:50.087661 13796 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:51:50.088794 13796 webserver.cc:460] Webserver started at http://127.10.225.129:42785/ using document root <none> and password file <none>
I20260806 23:51:50.089421 13796 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:50.089510 13796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:50.089723 13796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:50.092067 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "ba9f8ac583194a2fb0210c759d54eb16"
format_stamp: "Formatted at 2026-08-06 23:51:50 on dist-test-slave-nj21"
server_key: "629d3298b55229588e4f3153d2658acb"
server_key_iv: "34ef694541e70005fbace746c5c88911"
server_key_version: "encryptionkey@0"
I20260806 23:51:50.092795 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "ba9f8ac583194a2fb0210c759d54eb16"
format_stamp: "Formatted at 2026-08-06 23:51:50 on dist-test-slave-nj21"
server_key: "629d3298b55229588e4f3153d2658acb"
server_key_iv: "34ef694541e70005fbace746c5c88911"
server_key_version: "encryptionkey@0"
I20260806 23:51:50.097146 13796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:51:50.099880 13822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:50.101868 13796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:51:50.102115 13796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "ba9f8ac583194a2fb0210c759d54eb16"
format_stamp: "Formatted at 2026-08-06 23:51:50 on dist-test-slave-nj21"
server_key: "629d3298b55229588e4f3153d2658acb"
server_key_iv: "34ef694541e70005fbace746c5c88911"
server_key_version: "encryptionkey@0"
I20260806 23:51:50.102288 13796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:50.132223 13796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:50.136478 13796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:50.136744 13796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:50.138876 13796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:50.138952 13796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:50.139019 13796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:50.139047 13796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:51:50 dist-test-slave-nj21 krb5kdc[13608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:50.152290 13796 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:34361
I20260806 23:51:50.152362 13999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:34361 every 8 connection(s)
I20260806 23:51:50.153415 13796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:51:50.157086 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.140573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:37425 (local address 127.10.225.190:38301)
0806 23:51:50.140762 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.140767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.141965 (+ 1198us) server_negotiation.cc:407] Connection header received
0806 23:51:50.143738 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.143743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.143831 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.143949 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:50.146266 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.146940 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.148487 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.148786 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.148895 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.152662 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:50.152666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:50.152668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:50.152706 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:50.154512 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.155089 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.155092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.155093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.155180 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.155571 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.155574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.155576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.155834 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:50.155918 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.156803 (+ 885us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.156950 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 23:51:50.158279 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.141100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38301 (local address 127.10.225.129:37425)
0806 23:51:50.141735 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:50.141810 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:50.143334 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:50.144237 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:50.144249 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:50.144951 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:50.145955 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.145988 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.147169 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.147173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.148326 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.148352 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.148956 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.148961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.149153 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.149886 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:50.149907 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:50.152347 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:50.154653 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.154660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.154678 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.154984 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.155266 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.155270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.155272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.155450 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.156025 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:50.156034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:50.156425 (+ 391us) client_negotiation.cc:769] Sending connection context
0806 23:51:50.156726 (+ 301us) client_negotiation.cc:240] Negotiation successful
0806 23:51:50.157118 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":151,"threads_started":1}
I20260806 23:51:50.160198 14000 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38301
I20260806 23:51:50.160394 14000 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:50.160871 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 13796
I20260806 23:51:50.160993 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:51:50.161032 14000 heartbeater.cc:511] Master 127.10.225.190:38301 requested a full tablet report, sending...
I20260806 23:51:50.161322 11142 external_mini_cluster.cc:1428] Setting key 48b718b29f780372a4651b79f84fa0e1
I20260806 23:51:50.162823 13683 ts_manager.cc:194] Registered new tserver with Master: ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129:34361)
I20260806 23:51:50.164139 13683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:37425
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:51:50.175501 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.162946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38301 (local address 127.10.225.129:36537)
0806 23:51:50.164847 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:50.164861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:50.166861 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:50.167222 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:50.167225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:50.167478 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:50.167612 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.167619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.168770 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.168774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.169741 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.169755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.170347 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.170351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.170458 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.171084 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:50.171100 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:50.171615 (+ 515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:50.173401 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.173404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.173406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.173643 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.173992 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.173996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.173997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.174183 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.174924 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:50.174927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:50.175033 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 23:51:50.175139 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 23:51:50.175281 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"thread_start_us":95,"threads_started":1}
I20260806 23:51:50.175593 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.162994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:36537 (local address 127.10.225.190:38301)
0806 23:51:50.163120 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.163125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.164923 (+ 1798us) server_negotiation.cc:407] Connection header received
0806 23:51:50.167044 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.167047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.167097 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.167173 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:50.167880 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.168577 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.169935 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.170194 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.170307 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.171844 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:50.171847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:50.171849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:50.171873 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:50.173250 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.173733 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.173736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.173737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.173819 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.174397 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.174401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.174403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.174779 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:50.174896 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.175197 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.175392 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:51:50.192481 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38301
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:50.325169 14016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:50.325559 14016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:50.325634 14016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:50.330268 14016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:50.330348 14016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:50.330391 14016 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 23:51:50.330404 14016 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:51:50.330438 14016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:51:50.338068 14016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.225.190:38301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:50.339805 14016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:50.341188 14016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:50.354171 14029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:50.354202 14016 server_base.cc:1034] running on GCE node
W20260806 23:51:50.354151 14028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:50.354151 14035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:50.354977 14016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:50.356231 14016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:50.357468 14016 hybrid_clock.cc:648] HybridClock initialized: now 1786060310357434 us; error 47 us; skew 500 ppm
Aug 06 23:51:50 dist-test-slave-nj21 krb5kdc[13608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:50.361143 14016 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:51:50.362484 14016 webserver.cc:460] Webserver started at http://127.10.225.130:35017/ using document root <none> and password file <none>
I20260806 23:51:50.363211 14016 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:50.363286 14016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:50.363515 14016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:50.366631 14016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "9b402b105ce84188a922e83526b84b72"
format_stamp: "Formatted at 2026-08-06 23:51:50 on dist-test-slave-nj21"
server_key: "26af51baa812c666a56a4cb19331c348"
server_key_iv: "8878b318b4c18f007279677f318b06e0"
server_key_version: "encryptionkey@0"
I20260806 23:51:50.367410 14016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "9b402b105ce84188a922e83526b84b72"
format_stamp: "Formatted at 2026-08-06 23:51:50 on dist-test-slave-nj21"
server_key: "26af51baa812c666a56a4cb19331c348"
server_key_iv: "8878b318b4c18f007279677f318b06e0"
server_key_version: "encryptionkey@0"
I20260806 23:51:50.372013 14016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260806 23:51:50.374764 14047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:50.376451 14016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:51:50.376570 14016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "9b402b105ce84188a922e83526b84b72"
format_stamp: "Formatted at 2026-08-06 23:51:50 on dist-test-slave-nj21"
server_key: "26af51baa812c666a56a4cb19331c348"
server_key_iv: "8878b318b4c18f007279677f318b06e0"
server_key_version: "encryptionkey@0"
I20260806 23:51:50.376704 14016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:50.390218 14016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:50.393501 14016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:50.393695 14016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:50.395555 14016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:50.395623 14016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:50.395709 14016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:50.395735 14016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:51:50 dist-test-slave-nj21 krb5kdc[13608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:50.406409 14016 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:37535
I20260806 23:51:50.406798 14226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:37535 every 8 connection(s)
I20260806 23:51:50.407588 14016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:51:50.409868 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.396525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:35903 (local address 127.10.225.190:38301)
0806 23:51:50.396683 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.396688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.397628 (+ 940us) server_negotiation.cc:407] Connection header received
0806 23:51:50.398875 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.398879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.398941 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.399037 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:50.400442 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.401010 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.402665 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.403035 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.403142 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.405780 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:50.405784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:50.405787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:50.405825 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:50.407611 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.408317 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.408320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.408322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.408424 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.408782 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.408784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.408785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.408999 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:50.409079 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.409605 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.409722 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:51:50.410646 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.396940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38301 (local address 127.10.225.130:35903)
0806 23:51:50.397448 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:50.397510 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:50.398623 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:50.399227 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:50.399234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:50.399627 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:50.400301 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.400318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.401127 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.401131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.402546 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.402559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.403184 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.403188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.403378 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.404013 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:50.404030 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:50.405601 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:50.407759 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.407765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.407782 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.408125 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.408524 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.408527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.408529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.408703 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.409098 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:50.409103 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:50.409343 (+ 240us) client_negotiation.cc:769] Sending connection context
0806 23:51:50.409521 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 23:51:50.409788 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260806 23:51:50.411623 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 14016
I20260806 23:51:50.411741 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:51:50.411971 14227 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38301
I20260806 23:51:50.412009 11142 external_mini_cluster.cc:1428] Setting key 0c857b908238ec4c8f40669bb91be962
I20260806 23:51:50.412118 14227 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:50.412707 14227 heartbeater.cc:511] Master 127.10.225.190:38301 requested a full tablet report, sending...
I20260806 23:51:50.413712 13687 ts_manager.cc:194] Registered new tserver with Master: 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535)
I20260806 23:51:50.414415 13687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:35903
I20260806 23:51:50.422497 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.414698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:37325 (local address 127.10.225.190:38301)
0806 23:51:50.414802 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.414806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.414817 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:51:50.414855 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.414857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.414895 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.414969 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:50.415812 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.416488 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.417476 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.417708 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.417802 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.419219 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:50.419223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:50.419225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:50.419250 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:50.420754 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.421255 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.421258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.421259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.421344 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.421607 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.421610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.421612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.421858 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:50.421949 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.422210 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.422329 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 23:51:50.422523 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.414184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38301 (local address 127.10.225.130:37325)
0806 23:51:50.414484 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:50.414499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:50.414589 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:50.415086 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:50.415089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:50.415341 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:50.415468 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.415476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.416700 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.416705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.417345 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.417359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.417832 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.417836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.417941 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.418590 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:50.418604 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:50.419052 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:50.420914 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.420918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.420921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.421154 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.421435 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.421438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.421440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.421524 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.421954 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:50.421957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:50.422058 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 23:51:50.422145 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 23:51:50.422330 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:51:50.445088 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38301
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:50.584792 14241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:50.585103 14241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:50.585162 14241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:50.589987 14241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:50.590073 14241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:50.590121 14241 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 23:51:50.590154 14241 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:51:50.590215 14241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:51:50.596392 14241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.225.190:38301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:50.598220 14241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:50.599642 14241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:50.611837 14249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:50.611910 14241 server_base.cc:1034] running on GCE node
W20260806 23:51:50.612079 14252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:50.611935 14248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:50.612581 14241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:50.613263 14241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:50.614506 14241 hybrid_clock.cc:648] HybridClock initialized: now 1786060310614455 us; error 44 us; skew 500 ppm
Aug 06 23:51:50 dist-test-slave-nj21 krb5kdc[13608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:50.617834 14241 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:51:50.618851 14241 webserver.cc:460] Webserver started at http://127.10.225.131:44205/ using document root <none> and password file <none>
I20260806 23:51:50.619472 14241 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:50.619542 14241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:50.619822 14241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:50.622084 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "9d7b0f2de387401faa23d07c6692d8d7"
format_stamp: "Formatted at 2026-08-06 23:51:50 on dist-test-slave-nj21"
server_key: "91f1790af6ceb24a6ef5ff079b1f5edb"
server_key_iv: "3239fb2c3cd9af8ee347631513377432"
server_key_version: "encryptionkey@0"
I20260806 23:51:50.622656 14241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "9d7b0f2de387401faa23d07c6692d8d7"
format_stamp: "Formatted at 2026-08-06 23:51:50 on dist-test-slave-nj21"
server_key: "91f1790af6ceb24a6ef5ff079b1f5edb"
server_key_iv: "3239fb2c3cd9af8ee347631513377432"
server_key_version: "encryptionkey@0"
I20260806 23:51:50.626922 14241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 23:51:50.629874 14264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:50.631278 14241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:51:50.631407 14241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "9d7b0f2de387401faa23d07c6692d8d7"
format_stamp: "Formatted at 2026-08-06 23:51:50 on dist-test-slave-nj21"
server_key: "91f1790af6ceb24a6ef5ff079b1f5edb"
server_key_iv: "3239fb2c3cd9af8ee347631513377432"
server_key_version: "encryptionkey@0"
I20260806 23:51:50.631526 14241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:50.649397 14241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:50.652797 14241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:50.653007 14241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:50.654994 14241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:50.655107 14241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:50.655180 14241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:50.655212 14241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:51:50 dist-test-slave-nj21 krb5kdc[13608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060310, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:50.668780 14241 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:38093
I20260806 23:51:50.669044 14445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:38093 every 8 connection(s)
I20260806 23:51:50.669782 14241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:51:50.672273 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.655664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:36329 (local address 127.10.225.190:38301)
0806 23:51:50.655822 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.655827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.658100 (+ 2273us) server_negotiation.cc:407] Connection header received
0806 23:51:50.659739 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.659744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.659829 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.659973 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:50.661864 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.663016 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.664762 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.665005 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.665118 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.668401 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:50.668405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:50.668408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:50.668439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:50.669984 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.670651 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.670655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.670657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.670750 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.671145 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.671147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.671149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.671347 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:50.671422 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.672017 (+ 595us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.672134 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:51:50.673033 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.657111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38301 (local address 127.10.225.131:36329)
0806 23:51:50.657865 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:50.657930 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:50.659421 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:50.660268 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:50.660279 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:50.660805 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:50.661610 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.661637 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.663229 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.663232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.664604 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.664618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.665265 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.665289 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.665565 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.666472 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:50.666495 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:50.668163 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:50.670108 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.670115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.670131 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.670481 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.670868 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.670872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.670873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.671053 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.671430 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:50.671436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:50.671733 (+ 297us) client_negotiation.cc:769] Sending connection context
0806 23:51:50.671909 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 23:51:50.672196 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":188,"threads_started":1}
I20260806 23:51:50.674407 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 14241
I20260806 23:51:50.674407 14449 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38301
I20260806 23:51:50.674687 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:51:50.674718 14449 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:50.675173 11142 external_mini_cluster.cc:1428] Setting key bbdb5320dce4986044dfd52db13574f1
I20260806 23:51:50.675417 14449 heartbeater.cc:511] Master 127.10.225.190:38301 requested a full tablet report, sending...
I20260806 23:51:50.677629 13683 ts_manager.cc:194] Registered new tserver with Master: 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093)
I20260806 23:51:50.678498 13683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:36329
I20260806 23:51:50.686074 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.677102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:40097 (local address 127.10.225.190:38301)
0806 23:51:50.677517 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.677521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.677647 (+ 126us) server_negotiation.cc:407] Connection header received
0806 23:51:50.677735 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.677738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.677789 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.677889 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:50.678509 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.679431 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.680786 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.681035 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.681137 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.682716 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:50.682720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:50.682723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:50.682752 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:50.684370 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.684862 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.684866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.684868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.684950 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.685207 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.685211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.685212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.685420 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:50.685506 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.685773 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.685894 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
I20260806 23:51:50.686094 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.677027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38301 (local address 127.10.225.131:40097)
0806 23:51:50.677516 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:50.677532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:50.677638 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:50.677929 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:50.677933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:50.678161 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:50.678366 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.678380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.679646 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.679650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.680616 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.680632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.681173 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.681177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.681348 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.682061 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:50.682076 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:50.682515 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:50.684517 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.684521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.684524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.684762 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.685036 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:50.685039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:50.685041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:50.685123 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:50.685512 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:50.685515 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:50.685618 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 23:51:50.685708 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 23:51:50.685901 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":42,"thread_start_us":283,"threads_started":1}
I20260806 23:51:50.690697 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 23:51:50 dist-test-slave-nj21 krb5kdc[13608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:51:50 dist-test-slave-nj21 krb5kdc[13608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:50.720546 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.711707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.10.225.190:38301)
0806 23:51:50.711854 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.711858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.711872 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:51:50.711915 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.711918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.711961 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.712037 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:50.712870 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.713391 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.714237 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.714455 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.714539 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.717142 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:50.717145 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:50.717147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:50.717173 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:50.718508 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.718953 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.718956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.718958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.719044 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:50.719423 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:50.719426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:50.719427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:50.719757 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:50.719878 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.720291 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.720400 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:51:50.724506 13683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:51:50.727792 13683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:51:50.745891 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.739863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.10.225.129:34361)
0806 23:51:50.740266 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.740286 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.740897 (+ 611us) server_negotiation.cc:407] Connection header received
0806 23:51:50.740990 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.740996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.741176 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.741326 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:50.741730 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.742838 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.744437 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.745177 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.745301 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.745455 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.745627 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.745732 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260806 23:51:50.745891 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.738454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:34361 (local address 127.0.0.1:57900)
0806 23:51:50.740643 (+ 2189us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:50.740659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:50.740792 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:50.741468 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:50.741471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:50.741488 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:50.741577 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.741583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.742974 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.742978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.744249 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.744278 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.745301 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.745304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.745428 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.745471 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 23:51:50.745584 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 23:51:50.745728 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2112,"mutex_wait_us":296,"thread_start_us":136,"threads_started":1}
I20260806 23:51:50.746424 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.738976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.10.225.131:38093)
0806 23:51:50.739832 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.739855 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.739897 (+ 42us) server_negotiation.cc:407] Connection header received
0806 23:51:50.740227 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.740235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.740651 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.740844 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:50.741326 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.742213 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.744353 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.745190 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.745330 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.745461 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.746059 (+ 598us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.746266 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":108,"threads_started":1}
I20260806 23:51:50.746788 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.738585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:38093 (local address 127.0.0.1:38766)
0806 23:51:50.739749 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:50.739813 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:50.739979 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:50.741024 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:50.741031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:50.741054 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:50.741191 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.741199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.742429 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.742435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.744180 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.744192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.745529 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.745533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.745625 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.745648 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:51:50.745713 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:51:50.745782 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":66,"thread_start_us":258,"threads_started":1}
I20260806 23:51:50.748020 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.738452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:37535 (local address 127.0.0.1:50856)
0806 23:51:50.739746 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:50.739826 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:50.739982 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:50.743460 (+ 3478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:50.743463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:50.743485 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:50.743585 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.743592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.744959 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.744963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.746233 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:50.746245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.747557 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.747561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:50.747656 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.747731 (+ 75us) client_negotiation.cc:769] Sending connection context
0806 23:51:50.747803 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 23:51:50.747881 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":96,"threads_started":1}
I20260806 23:51:50.748443 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:50.739403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.10.225.130:37535)
0806 23:51:50.742752 (+ 3349us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:50.742765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:50.742793 (+ 28us) server_negotiation.cc:407] Connection header received
0806 23:51:50.742867 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:50.742872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:50.743045 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:50.743203 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:50.743759 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.744824 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.746388 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:50.747230 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:50.747363 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:50.747488 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:50.748197 (+ 709us) server_negotiation.cc:299] Negotiation successful
0806 23:51:50.748300 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3209,"thread_start_us":128,"threads_started":1}
I20260806 23:51:50.749365 14346 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce52166157942abbe50938afaaf972f (DEFAULT_TABLE table=test-table [id=eb30a6edec004e12ad06cd3149c58c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:50.749353 13900 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce52166157942abbe50938afaaf972f (DEFAULT_TABLE table=test-table [id=eb30a6edec004e12ad06cd3149c58c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:50.750349 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce52166157942abbe50938afaaf972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:50.750355 14346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce52166157942abbe50938afaaf972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:50.754694 14126 tablet_service.cc:1467] Processing CreateTablet for tablet 9ce52166157942abbe50938afaaf972f (DEFAULT_TABLE table=test-table [id=eb30a6edec004e12ad06cd3149c58c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:50.755918 14126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ce52166157942abbe50938afaaf972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:50.758983 14484 tablet_bootstrap.cc:492] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: Bootstrap starting.
I20260806 23:51:50.760023 14485 tablet_bootstrap.cc:492] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: Bootstrap starting.
I20260806 23:51:50.762215 14485 tablet_bootstrap.cc:654] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:50.763154 14485 log.cc:824] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:50.763965 14484 tablet_bootstrap.cc:654] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:50.764873 14484 log.cc:824] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:50.764979 14492 tablet_bootstrap.cc:492] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: Bootstrap starting.
I20260806 23:51:50.765496 14485 tablet_bootstrap.cc:492] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: No bootstrap required, opened a new log
I20260806 23:51:50.765708 14485 ts_tablet_manager.cc:1397] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:51:50.766918 14484 tablet_bootstrap.cc:492] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: No bootstrap required, opened a new log
I20260806 23:51:50.767002 14492 tablet_bootstrap.cc:654] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:50.767114 14484 ts_tablet_manager.cc:1397] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260806 23:51:50.767760 14492 log.cc:824] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: Log is configured to *not* fsync() on all Append() calls
I20260806 23:51:50.769400 14492 tablet_bootstrap.cc:492] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: No bootstrap required, opened a new log
I20260806 23:51:50.769595 14492 ts_tablet_manager.cc:1397] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 23:51:50.769870 14485 raft_consensus.cc:348] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:50.770157 14485 raft_consensus.cc:374] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:50.770211 14485 raft_consensus.cc:729] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7b0f2de387401faa23d07c6692d8d7, State: Initialized, Role: FOLLOWER
I20260806 23:51:50.770696 14485 consensus_queue.cc:260] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:50.771818 14485 ts_tablet_manager.cc:1428] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 23:51:50.771823 14484 raft_consensus.cc:348] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:50.772087 14484 raft_consensus.cc:374] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:50.772174 14484 raft_consensus.cc:729] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9f8ac583194a2fb0210c759d54eb16, State: Initialized, Role: FOLLOWER
I20260806 23:51:50.772254 14449 heartbeater.cc:503] Master 127.10.225.190:38301 was elected leader, sending a full tablet report...
I20260806 23:51:50.772627 14484 consensus_queue.cc:260] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:50.773284 14000 heartbeater.cc:503] Master 127.10.225.190:38301 was elected leader, sending a full tablet report...
I20260806 23:51:50.773293 14492 raft_consensus.cc:348] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:50.773542 14492 raft_consensus.cc:374] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:50.773592 14492 raft_consensus.cc:729] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b402b105ce84188a922e83526b84b72, State: Initialized, Role: FOLLOWER
I20260806 23:51:50.773615 14484 ts_tablet_manager.cc:1428] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:51:50.774068 14492 consensus_queue.cc:260] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:50.775002 14227 heartbeater.cc:503] Master 127.10.225.190:38301 was elected leader, sending a full tablet report...
I20260806 23:51:50.775346 14492 ts_tablet_manager.cc:1428] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260806 23:51:50.905166 14004 tablet.cc:2369] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:51:50.908926 14229 tablet.cc:2369] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:51:50.921046 14453 tablet.cc:2369] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:51:51.072049 14496 raft_consensus.cc:482] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:51:51.072387 14496 raft_consensus.cc:504] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.073693 14496 leader_election.cc:288] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535), 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093)
I20260806 23:51:51.078387 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.074033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:37249 (local address 127.10.225.130:37535)
0806 23:51:51.074202 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:51.074208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:51.074714 (+ 506us) server_negotiation.cc:407] Connection header received
0806 23:51:51.074904 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:51.074906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:51.074957 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:51.075032 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:51.075374 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.075989 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.077215 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.077643 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.077771 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.077848 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:51.078097 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 23:51:51.078216 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:51:51.078410 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.073931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:37535 (local address 127.10.225.129:37249)
0806 23:51:51.074542 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:51.074566 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:51.074773 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:51.075061 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:51.075064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:51.075082 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:51.075240 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.075250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.076109 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.076113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.077087 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.077098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.077771 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.077775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.077876 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.077904 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 23:51:51.078081 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 23:51:51.078201 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":47,"thread_start_us":140,"threads_started":1}
I20260806 23:51:51.078612 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.074211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:39541 (local address 127.10.225.131:38093)
0806 23:51:51.074531 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:51.074537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:51.074611 (+ 74us) server_negotiation.cc:407] Connection header received
0806 23:51:51.074780 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:51.074783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:51.074828 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:51.074898 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:51.075166 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.075937 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.077349 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.077884 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.078063 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.078124 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:51.078311 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 23:51:51.078396 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 23:51:51.078737 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.074006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:38093 (local address 127.10.225.129:39541)
0806 23:51:51.074516 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:51.074532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:51.074651 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:51.074931 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:51.074933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:51.074952 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:51.075069 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.075074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.076052 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.076056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.077098 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.077109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.078063 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.078068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.078191 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.078213 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:51:51.078283 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 23:51:51.078366 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":73,"thread_start_us":109,"threads_started":1}
I20260806 23:51:51.079440 14158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce52166157942abbe50938afaaf972f" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72" is_pre_election: true
I20260806 23:51:51.079710 14381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce52166157942abbe50938afaaf972f" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7b0f2de387401faa23d07c6692d8d7" is_pre_election: true
I20260806 23:51:51.079813 14158 raft_consensus.cc:2454] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 0.
I20260806 23:51:51.080114 14381 raft_consensus.cc:2454] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 0.
I20260806 23:51:51.080473 13826 leader_election.cc:302] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b402b105ce84188a922e83526b84b72, ba9f8ac583194a2fb0210c759d54eb16; no voters:
I20260806 23:51:51.080826 14496 raft_consensus.cc:2781] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:51:51.080940 14496 raft_consensus.cc:482] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:51:51.080994 14496 raft_consensus.cc:3037] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:51.082408 14496 raft_consensus.cc:504] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.082872 14496 leader_election.cc:288] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 election: Requested vote from peers 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535), 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093)
I20260806 23:51:51.083300 14381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce52166157942abbe50938afaaf972f" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7b0f2de387401faa23d07c6692d8d7"
I20260806 23:51:51.083434 14381 raft_consensus.cc:3037] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:51.083427 14158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ce52166157942abbe50938afaaf972f" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72"
I20260806 23:51:51.083585 14158 raft_consensus.cc:3037] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:51.084893 14158 raft_consensus.cc:2454] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 1.
I20260806 23:51:51.085161 14381 raft_consensus.cc:2454] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 1.
I20260806 23:51:51.085249 13826 leader_election.cc:302] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b402b105ce84188a922e83526b84b72, ba9f8ac583194a2fb0210c759d54eb16; no voters:
I20260806 23:51:51.085475 14496 raft_consensus.cc:2781] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:51.085870 14496 raft_consensus.cc:686] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 1 LEADER]: Becoming Leader. State: Replica: ba9f8ac583194a2fb0210c759d54eb16, State: Running, Role: LEADER
I20260806 23:51:51.086282 14496 consensus_queue.cc:237] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.090061 13683 catalog_manager.cc:5495] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129). New cstate: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } health_report { overall_health: UNKNOWN } } }
Aug 06 23:51:51 dist-test-slave-nj21 krb5kdc[13608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060309, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:51.140461 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.129931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.10.225.190:38301)
0806 23:51:51.130079 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:51.130083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:51.130260 (+ 177us) server_negotiation.cc:407] Connection header received
0806 23:51:51.130539 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:51.130543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:51.130584 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:51.130648 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:51.131457 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.132057 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.132869 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.133091 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.133171 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.136322 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:51.136325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:51.136327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:51.136350 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:51.138140 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:51.138739 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:51.138743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:51.138744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:51.138857 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:51.139296 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:51.139299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:51.139300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:51.139582 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:51.139739 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:51.140159 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 23:51:51.140334 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":65}
I20260806 23:51:51.140514 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.129818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38301 (local address 127.0.0.1:39340)
0806 23:51:51.130170 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:51.130184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:51.130292 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:51.130757 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:51.130760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:51.131197 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:51.131312 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.131319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.132179 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.132182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.132746 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.132754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.133238 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.133241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.133309 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.133984 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:51.134009 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:51.136133 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:51.138294 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:51.138306 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:51.138312 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:51.138597 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:51.139041 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:51.139044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:51.139046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:51.139157 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:51.139774 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:51.139784 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:51.139931 (+ 147us) client_negotiation.cc:769] Sending connection context
0806 23:51:51.140055 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 23:51:51.140260 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":126,"threads_started":1}
I20260806 23:51:51.146968 13683 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:39340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 23:51:51.147876 13683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:51:51.153154 13900 tablet_service.cc:1467] Processing CreateTablet for tablet 0b89332266964a1db77e997595f6853f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5a9e3e8e64544c6b4d15c0d1a725786]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:51:51.153365 14126 tablet_service.cc:1467] Processing CreateTablet for tablet 0b89332266964a1db77e997595f6853f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5a9e3e8e64544c6b4d15c0d1a725786]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:51:51.153357 14346 tablet_service.cc:1467] Processing CreateTablet for tablet 0b89332266964a1db77e997595f6853f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5a9e3e8e64544c6b4d15c0d1a725786]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:51:51.153936 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b89332266964a1db77e997595f6853f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:51.154042 14346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b89332266964a1db77e997595f6853f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:51.154042 14126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b89332266964a1db77e997595f6853f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:51.157200 14485 tablet_bootstrap.cc:492] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: Bootstrap starting.
I20260806 23:51:51.158231 14484 tablet_bootstrap.cc:492] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: Bootstrap starting.
I20260806 23:51:51.158349 14485 tablet_bootstrap.cc:654] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:51.159238 14484 tablet_bootstrap.cc:654] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:51.159756 14485 tablet_bootstrap.cc:492] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: No bootstrap required, opened a new log
I20260806 23:51:51.159855 14485 ts_tablet_manager.cc:1397] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:51:51.160329 14485 raft_consensus.cc:348] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.160463 14485 raft_consensus.cc:374] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:51.160470 14484 tablet_bootstrap.cc:492] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: No bootstrap required, opened a new log
I20260806 23:51:51.160499 14485 raft_consensus.cc:729] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7b0f2de387401faa23d07c6692d8d7, State: Initialized, Role: FOLLOWER
I20260806 23:51:51.160538 14484 ts_tablet_manager.cc:1397] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260806 23:51:51.160903 14485 consensus_queue.cc:260] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.160960 14484 raft_consensus.cc:348] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.161067 14484 raft_consensus.cc:374] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:51.161098 14484 raft_consensus.cc:729] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9f8ac583194a2fb0210c759d54eb16, State: Initialized, Role: FOLLOWER
I20260806 23:51:51.161213 14484 consensus_queue.cc:260] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.161190 14495 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.161329 14495 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:51.161459 14496 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.161571 14485 ts_tablet_manager.cc:1428] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:51:51.161592 14496 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:51.161473 14484 ts_tablet_manager.cc:1428] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 23:51:51.161927 14492 tablet_bootstrap.cc:492] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: Bootstrap starting.
I20260806 23:51:51.162959 14492 tablet_bootstrap.cc:654] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:51.164345 14492 tablet_bootstrap.cc:492] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: No bootstrap required, opened a new log
I20260806 23:51:51.164436 14492 ts_tablet_manager.cc:1397] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:51:51.164945 14492 raft_consensus.cc:348] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.165064 14492 raft_consensus.cc:374] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:51.165099 14492 raft_consensus.cc:729] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b402b105ce84188a922e83526b84b72, State: Initialized, Role: FOLLOWER
I20260806 23:51:51.165227 14492 consensus_queue.cc:260] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.165499 14492 ts_tablet_manager.cc:1428] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:51.165518 14497 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.165668 14497 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:51.211783 14496 raft_consensus.cc:482] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:51:51.211923 14496 raft_consensus.cc:504] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.212340 14496 leader_election.cc:288] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535), 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093)
I20260806 23:51:51.212857 14381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b89332266964a1db77e997595f6853f" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7b0f2de387401faa23d07c6692d8d7" is_pre_election: true
I20260806 23:51:51.212841 14158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b89332266964a1db77e997595f6853f" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72" is_pre_election: true
I20260806 23:51:51.213006 14158 raft_consensus.cc:2454] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 0.
I20260806 23:51:51.213006 14381 raft_consensus.cc:2454] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 0.
I20260806 23:51:51.213300 13826 leader_election.cc:302] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b402b105ce84188a922e83526b84b72, ba9f8ac583194a2fb0210c759d54eb16; no voters:
I20260806 23:51:51.213465 14496 raft_consensus.cc:2781] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:51:51.213533 14496 raft_consensus.cc:482] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:51:51.213565 14496 raft_consensus.cc:3037] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:51.214524 14496 raft_consensus.cc:504] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.214836 14496 leader_election.cc:288] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 election: Requested vote from peers 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535), 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093)
I20260806 23:51:51.215233 14381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b89332266964a1db77e997595f6853f" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7b0f2de387401faa23d07c6692d8d7"
I20260806 23:51:51.215359 14381 raft_consensus.cc:3037] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:51.215534 14158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b89332266964a1db77e997595f6853f" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72"
I20260806 23:51:51.215757 14158 raft_consensus.cc:3037] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:51.216418 14381 raft_consensus.cc:2454] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 1.
I20260806 23:51:51.216702 13827 leader_election.cc:302] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7b0f2de387401faa23d07c6692d8d7, ba9f8ac583194a2fb0210c759d54eb16; no voters:
I20260806 23:51:51.216862 14496 raft_consensus.cc:2781] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:51.216959 14496 raft_consensus.cc:686] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [term 1 LEADER]: Becoming Leader. State: Replica: ba9f8ac583194a2fb0210c759d54eb16, State: Running, Role: LEADER
I20260806 23:51:51.216970 14158 raft_consensus.cc:2454] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 1.
I20260806 23:51:51.217110 14496 consensus_queue.cc:237] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.217880 14501 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: TxnStatusTablet state changed. Reason: New leader ba9f8ac583194a2fb0210c759d54eb16. Latest consensus state: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.218057 14501 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:51:51.218466 14519 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 23:51:51.218721 14519 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 23:51:51.219362 13687 catalog_manager.cc:5495] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129). New cstate: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } health_report { overall_health: UNKNOWN } } }
I20260806 23:51:51.251242 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.246425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.10.225.129:34361)
0806 23:51:51.246566 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:51.246571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:51.246822 (+ 251us) server_negotiation.cc:407] Connection header received
0806 23:51:51.247137 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:51.247140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:51.247227 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:51.247345 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:51.247821 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.248361 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.249641 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.249933 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.250027 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.250387 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:51.250867 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:51.250956 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:51.251056 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 23:51:51.251114 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:51:51.251281 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.246268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:34361 (local address 127.0.0.1:57916)
0806 23:51:51.246693 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:51.246722 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:51.246921 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:51.247383 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:51.247386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:51.247395 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:51:51.247560 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.247567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.248501 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.248505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.249501 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.249514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.250046 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.250049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.250136 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.250207 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:51:51.250954 (+ 747us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:51:51.250966 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 23:51:51.251034 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 23:51:51.251127 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260806 23:51:51.257192 14381 raft_consensus.cc:1261] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Refusing update from remote peer ba9f8ac583194a2fb0210c759d54eb16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:51.257767 14158 raft_consensus.cc:1261] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Refusing update from remote peer ba9f8ac583194a2fb0210c759d54eb16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:51.258066 14501 consensus_queue.cc:1035] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:51.258610 14501 consensus_queue.cc:1035] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:51.262984 14495 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: TxnStatusTablet state changed. Reason: New leader ba9f8ac583194a2fb0210c759d54eb16. Latest consensus state: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.263144 14495 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:51.263345 14497 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: TxnStatusTablet state changed. Reason: New leader ba9f8ac583194a2fb0210c759d54eb16. Latest consensus state: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.263492 14497 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:51.268327 14501 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.268494 14501 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:51:51.269191 14495 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.269375 14495 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P 9d7b0f2de387401faa23d07c6692d8d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:51.271742 14523 mvcc.cc:205] Tried to move back new op lower bound from 7315703034900713472 to 7315703034746712064. Current Snapshot: MvccSnapshot[applied={T|T < 7315703034900713472}]
I20260806 23:51:51.272231 14522 mvcc.cc:205] Tried to move back new op lower bound from 7315703034900713472 to 7315703034746712064. Current Snapshot: MvccSnapshot[applied={T|T < 7315703034900713472 or (T in {7315703034900713472})}]
I20260806 23:51:51.272712 14496 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.272848 14496 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P ba9f8ac583194a2fb0210c759d54eb16: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:51:51.274644 14524 mvcc.cc:205] Tried to move back new op lower bound from 7315703034900713472 to 7315703034746712064. Current Snapshot: MvccSnapshot[applied={T|T < 7315703034900713472}]
I20260806 23:51:51.275148 14497 tablet_replica.cc:439] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } } }
I20260806 23:51:51.275285 14497 tablet_replica.cc:442] T 0b89332266964a1db77e997595f6853f P 9b402b105ce84188a922e83526b84b72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:51:51.282020 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.276419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.10.225.129:34361)
0806 23:51:51.276565 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:51.276571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:51.277863 (+ 1292us) server_negotiation.cc:407] Connection header received
0806 23:51:51.277996 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:51.278000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:51.278068 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:51.278233 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:51.278483 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.279372 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.280434 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.280787 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.280909 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.281346 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:51.281460 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:51.281533 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:51.281787 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 23:51:51.281876 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 23:51:51.294111 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.290436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:35999 (local address 127.10.225.129:34361)
0806 23:51:51.290549 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:51.290553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:51.290713 (+ 160us) server_negotiation.cc:407] Connection header received
0806 23:51:51.290837 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:51.290841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:51.290884 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:51.290945 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:51.291174 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.291708 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.292969 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.293185 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.293269 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.293659 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:51.293740 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:51.293804 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:51.293913 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 23:51:51.293966 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":32}
I20260806 23:51:51.294159 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:51.290373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:34361 (local address 127.10.225.129:35999)
0806 23:51:51.290638 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:51.290650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:51.290757 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:51.290964 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:51.290966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:51.290973 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:51:51.291086 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.291092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.291820 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.291824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.292804 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:51.292816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:51.293317 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:51.293321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:51.293412 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:51.293467 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:51:51.293830 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:51:51.293837 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 23:51:51.293901 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 23:51:51.293988 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
I20260806 23:51:51.302244 14158 raft_consensus.cc:1261] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Refusing update from remote peer ba9f8ac583194a2fb0210c759d54eb16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:51.302376 14381 raft_consensus.cc:1261] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Refusing update from remote peer ba9f8ac583194a2fb0210c759d54eb16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:51.302772 14496 consensus_queue.cc:1035] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:51.303074 14533 consensus_queue.cc:1035] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:51.352010 13687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:51.352192 13687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:51.354743 13687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: Sending DeleteTablet for 3 replicas of tablet 9ce52166157942abbe50938afaaf972f
I20260806 23:51:51.355545 13898 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce52166157942abbe50938afaaf972f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:51 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:57900
I20260806 23:51:51.355890 14126 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce52166157942abbe50938afaaf972f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:51 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:50856
I20260806 23:51:51.356388 14549 tablet_replica.cc:330] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: stopping tablet replica
I20260806 23:51:51.356287 13687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 23:51:51.356600 14346 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ce52166157942abbe50938afaaf972f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:51 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:38766
W20260806 23:51:51.356724 13687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:51:51.357064 14549 raft_consensus.cc:2229] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:51:51.357095 14553 tablet_replica.cc:330] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: stopping tablet replica
I20260806 23:51:51.357590 14549 raft_consensus.cc:2258] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:51.357380 14553 raft_consensus.cc:2229] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:51.358538 14553 raft_consensus.cc:2258] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:51.359093 14549 ts_tablet_manager.cc:1905] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:51.362382 14126 tablet_service.cc:1467] Processing CreateTablet for tablet 91207905ab324c499f16cfe001f83c77 (DEFAULT_TABLE table=test-table [id=dbffd0647e0b450993f30db405012339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:51.362488 14549 ts_tablet_manager.cc:1918] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:51:51.362572 14549 log.cc:1196] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/9ce52166157942abbe50938afaaf972f
I20260806 23:51:51.362550 14346 tablet_service.cc:1467] Processing CreateTablet for tablet 91207905ab324c499f16cfe001f83c77 (DEFAULT_TABLE table=test-table [id=dbffd0647e0b450993f30db405012339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:51.362713 14126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91207905ab324c499f16cfe001f83c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:51.362838 14346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91207905ab324c499f16cfe001f83c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:51.362910 14549 ts_tablet_manager.cc:1939] T 9ce52166157942abbe50938afaaf972f P ba9f8ac583194a2fb0210c759d54eb16: Deleting consensus metadata
I20260806 23:51:51.363831 14552 tablet_replica.cc:330] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: stopping tablet replica
I20260806 23:51:51.364152 14552 raft_consensus.cc:2229] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:51.364373 14552 raft_consensus.cc:2258] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:51.365267 14492 tablet_bootstrap.cc:492] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: Bootstrap starting.
I20260806 23:51:51.365933 13659 catalog_manager.cc:4841] TS ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129:34361): tablet 9ce52166157942abbe50938afaaf972f (table test-table [id=eb30a6edec004e12ad06cd3149c58c38]) successfully deleted
I20260806 23:51:51.366201 14492 tablet_bootstrap.cc:654] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:51.366649 13898 tablet_service.cc:1467] Processing CreateTablet for tablet 91207905ab324c499f16cfe001f83c77 (DEFAULT_TABLE table=test-table [id=dbffd0647e0b450993f30db405012339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:51.366947 13898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91207905ab324c499f16cfe001f83c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:51.367481 14492 tablet_bootstrap.cc:492] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: No bootstrap required, opened a new log
I20260806 23:51:51.367576 14492 ts_tablet_manager.cc:1397] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 23:51:51.368105 14492 raft_consensus.cc:348] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.368229 14492 raft_consensus.cc:374] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:51.368268 14492 raft_consensus.cc:729] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b402b105ce84188a922e83526b84b72, State: Initialized, Role: FOLLOWER
I20260806 23:51:51.368664 14492 consensus_queue.cc:260] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.369527 14492 ts_tablet_manager.cc:1428] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 23:51:51.369782 14484 tablet_bootstrap.cc:492] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: Bootstrap starting.
I20260806 23:51:51.370445 14553 ts_tablet_manager.cc:1905] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:51.370832 14552 ts_tablet_manager.cc:1905] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:51.370904 14484 tablet_bootstrap.cc:654] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:51.371596 14485 tablet_bootstrap.cc:492] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: Bootstrap starting.
I20260806 23:51:51.372622 14485 tablet_bootstrap.cc:654] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:51.373773 14553 ts_tablet_manager.cc:1918] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:51:51.373875 14553 log.cc:1196] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/9ce52166157942abbe50938afaaf972f
I20260806 23:51:51.374141 14553 ts_tablet_manager.cc:1939] T 9ce52166157942abbe50938afaaf972f P 9d7b0f2de387401faa23d07c6692d8d7: Deleting consensus metadata
I20260806 23:51:51.375391 14484 tablet_bootstrap.cc:492] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: No bootstrap required, opened a new log
I20260806 23:51:51.375497 14484 ts_tablet_manager.cc:1397] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260806 23:51:51.375972 14484 raft_consensus.cc:348] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.376096 14484 raft_consensus.cc:374] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:51.376128 14484 raft_consensus.cc:729] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9f8ac583194a2fb0210c759d54eb16, State: Initialized, Role: FOLLOWER
I20260806 23:51:51.376289 14484 consensus_queue.cc:260] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.376591 14484 ts_tablet_manager.cc:1428] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:51.377408 14552 ts_tablet_manager.cc:1918] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:51:51.377460 14485 tablet_bootstrap.cc:492] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: No bootstrap required, opened a new log
I20260806 23:51:51.377513 14552 log.cc:1196] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/9ce52166157942abbe50938afaaf972f
I20260806 23:51:51.377552 14485 ts_tablet_manager.cc:1397] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 23:51:51.377820 14552 ts_tablet_manager.cc:1939] T 9ce52166157942abbe50938afaaf972f P 9b402b105ce84188a922e83526b84b72: Deleting consensus metadata
I20260806 23:51:51.378089 14485 raft_consensus.cc:348] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.378214 14485 raft_consensus.cc:374] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:51.378257 14485 raft_consensus.cc:729] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7b0f2de387401faa23d07c6692d8d7, State: Initialized, Role: FOLLOWER
I20260806 23:51:51.378382 14485 consensus_queue.cc:260] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.378671 14485 ts_tablet_manager.cc:1428] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:51.379555 13656 catalog_manager.cc:4841] TS 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535): tablet 9ce52166157942abbe50938afaaf972f (table test-table [id=eb30a6edec004e12ad06cd3149c58c38]) successfully deleted
I20260806 23:51:51.379873 13657 catalog_manager.cc:4841] TS 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093): tablet 9ce52166157942abbe50938afaaf972f (table test-table [id=eb30a6edec004e12ad06cd3149c58c38]) successfully deleted
I20260806 23:51:51.513171 14547 raft_consensus.cc:482] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:51:51.513330 14547 raft_consensus.cc:504] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.513816 14547 leader_election.cc:288] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535), 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093)
I20260806 23:51:51.514364 14379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91207905ab324c499f16cfe001f83c77" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7b0f2de387401faa23d07c6692d8d7" is_pre_election: true
I20260806 23:51:51.514351 14158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91207905ab324c499f16cfe001f83c77" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72" is_pre_election: true
I20260806 23:51:51.514528 14158 raft_consensus.cc:2454] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 0.
I20260806 23:51:51.514520 14379 raft_consensus.cc:2454] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 0.
I20260806 23:51:51.514901 13827 leader_election.cc:302] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9b402b105ce84188a922e83526b84b72, 9d7b0f2de387401faa23d07c6692d8d7, ba9f8ac583194a2fb0210c759d54eb16; no voters:
I20260806 23:51:51.515094 14547 raft_consensus.cc:2781] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:51:51.515158 14547 raft_consensus.cc:482] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:51:51.515190 14547 raft_consensus.cc:3037] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:51.516283 14547 raft_consensus.cc:504] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.516633 14547 leader_election.cc:288] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 election: Requested vote from peers 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535), 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093)
I20260806 23:51:51.517123 14158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91207905ab324c499f16cfe001f83c77" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72"
I20260806 23:51:51.517243 14158 raft_consensus.cc:3037] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:51.517235 14379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91207905ab324c499f16cfe001f83c77" candidate_uuid: "ba9f8ac583194a2fb0210c759d54eb16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7b0f2de387401faa23d07c6692d8d7"
I20260806 23:51:51.517465 14379 raft_consensus.cc:3037] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:51.518211 14158 raft_consensus.cc:2454] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 1.
I20260806 23:51:51.518501 13826 leader_election.cc:302] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b402b105ce84188a922e83526b84b72, ba9f8ac583194a2fb0210c759d54eb16; no voters:
I20260806 23:51:51.518662 14547 raft_consensus.cc:2781] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:51.518750 14547 raft_consensus.cc:686] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 1 LEADER]: Becoming Leader. State: Replica: ba9f8ac583194a2fb0210c759d54eb16, State: Running, Role: LEADER
I20260806 23:51:51.518895 14547 consensus_queue.cc:237] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:51.519227 14379 raft_consensus.cc:2454] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9f8ac583194a2fb0210c759d54eb16 in term 1.
I20260806 23:51:51.520978 13687 catalog_manager.cc:5495] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129). New cstate: current_term: 1 leader_uuid: "ba9f8ac583194a2fb0210c759d54eb16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } health_report { overall_health: UNKNOWN } } }
I20260806 23:51:51.541060 14379 raft_consensus.cc:1261] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Refusing update from remote peer ba9f8ac583194a2fb0210c759d54eb16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:51.541355 14156 raft_consensus.cc:1261] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Refusing update from remote peer ba9f8ac583194a2fb0210c759d54eb16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:51.541651 14501 consensus_queue.cc:1035] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:51.542358 14496 consensus_queue.cc:1035] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 23:51:51.559520 14522 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 23:51:53.120115 13683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:53.120350 13683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:53.123651 13683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: Sending DeleteTablet for 3 replicas of tablet 91207905ab324c499f16cfe001f83c77
I20260806 23:51:53.124403 13898 tablet_service.cc:1514] Processing DeleteTablet for tablet 91207905ab324c499f16cfe001f83c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:53 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:57900
I20260806 23:51:53.124507 14126 tablet_service.cc:1514] Processing DeleteTablet for tablet 91207905ab324c499f16cfe001f83c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:53 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:50856
I20260806 23:51:53.124595 14346 tablet_service.cc:1514] Processing DeleteTablet for tablet 91207905ab324c499f16cfe001f83c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:53 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:38766
I20260806 23:51:53.124853 14607 tablet_replica.cc:330] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: stopping tablet replica
I20260806 23:51:53.124943 14606 tablet_replica.cc:330] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: stopping tablet replica
I20260806 23:51:53.124975 14608 tablet_replica.cc:330] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: stopping tablet replica
I20260806 23:51:53.125070 14607 raft_consensus.cc:2229] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:53.125082 14606 raft_consensus.cc:2229] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:51:53.125180 14608 raft_consensus.cc:2229] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:53.125270 14607 raft_consensus.cc:2258] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:53.125286 14606 raft_consensus.cc:2258] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:53.125893 14608 raft_consensus.cc:2258] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:53.125976 14606 ts_tablet_manager.cc:1905] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:53.125976 14607 ts_tablet_manager.cc:1905] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:53.128835 14607 ts_tablet_manager.cc:1918] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:51:53.128835 14606 ts_tablet_manager.cc:1918] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:51:53.128964 14607 log.cc:1196] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/91207905ab324c499f16cfe001f83c77
I20260806 23:51:53.128919 14606 log.cc:1196] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/91207905ab324c499f16cfe001f83c77
I20260806 23:51:53.129256 14607 ts_tablet_manager.cc:1939] T 91207905ab324c499f16cfe001f83c77 P 9b402b105ce84188a922e83526b84b72: Deleting consensus metadata
I20260806 23:51:53.129345 14606 ts_tablet_manager.cc:1939] T 91207905ab324c499f16cfe001f83c77 P ba9f8ac583194a2fb0210c759d54eb16: Deleting consensus metadata
I20260806 23:51:53.129489 14608 ts_tablet_manager.cc:1905] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:53.130133 13659 catalog_manager.cc:4841] TS ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129:34361): tablet 91207905ab324c499f16cfe001f83c77 (table test-table [id=dbffd0647e0b450993f30db405012339]) successfully deleted
I20260806 23:51:53.130350 13656 catalog_manager.cc:4841] TS 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535): tablet 91207905ab324c499f16cfe001f83c77 (table test-table [id=dbffd0647e0b450993f30db405012339]) successfully deleted
I20260806 23:51:53.131819 14608 ts_tablet_manager.cc:1918] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:51:53.131963 14608 log.cc:1196] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/91207905ab324c499f16cfe001f83c77
I20260806 23:51:53.132248 14608 ts_tablet_manager.cc:1939] T 91207905ab324c499f16cfe001f83c77 P 9d7b0f2de387401faa23d07c6692d8d7: Deleting consensus metadata
I20260806 23:51:53.133002 13657 catalog_manager.cc:4841] TS 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093): tablet 91207905ab324c499f16cfe001f83c77 (table test-table [id=dbffd0647e0b450993f30db405012339]) successfully deleted
Aug 06 23:51:53 dist-test-slave-nj21 krb5kdc[13608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:51:53 dist-test-slave-nj21 krb5kdc[13608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:53.158270 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.148160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.10.225.190:38301)
0806 23:51:53.148472 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:53.148480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:53.148497 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:51:53.148590 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:53.148592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:53.148636 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:53.148715 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:53.149912 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.150480 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.151555 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.151933 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.152047 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.154812 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:53.154815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:53.154818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:53.154849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:53.156592 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:53.157013 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:53.157016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:53.157018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:53.157100 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:53.157354 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:53.157357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:53.157359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:53.157577 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:53.157725 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:53.157964 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 23:51:53.158098 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":123,"threads_started":1}
I20260806 23:51:53.161818 13683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:51:53.162355 13683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:51:53.168121 13898 tablet_service.cc:1467] Processing CreateTablet for tablet 77e928a72b5841f388cdb37c4b23501a (DEFAULT_TABLE table=test-table [id=62ac6f828b7543578c1d18997b4a8683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:53.168335 14346 tablet_service.cc:1467] Processing CreateTablet for tablet 77e928a72b5841f388cdb37c4b23501a (DEFAULT_TABLE table=test-table [id=62ac6f828b7543578c1d18997b4a8683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:53.168498 13898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e928a72b5841f388cdb37c4b23501a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:53.168473 14126 tablet_service.cc:1467] Processing CreateTablet for tablet 77e928a72b5841f388cdb37c4b23501a (DEFAULT_TABLE table=test-table [id=62ac6f828b7543578c1d18997b4a8683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:53.168622 14346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e928a72b5841f388cdb37c4b23501a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:53.168718 14126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e928a72b5841f388cdb37c4b23501a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:53.171056 14621 tablet_bootstrap.cc:492] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: Bootstrap starting.
I20260806 23:51:53.171537 14622 tablet_bootstrap.cc:492] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: Bootstrap starting.
I20260806 23:51:53.171785 14623 tablet_bootstrap.cc:492] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: Bootstrap starting.
I20260806 23:51:53.172220 14621 tablet_bootstrap.cc:654] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:53.172448 14622 tablet_bootstrap.cc:654] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:53.172578 14623 tablet_bootstrap.cc:654] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:53.173287 14621 tablet_bootstrap.cc:492] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: No bootstrap required, opened a new log
I20260806 23:51:53.173374 14621 ts_tablet_manager.cc:1397] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 23:51:53.173759 14621 raft_consensus.cc:348] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.173883 14621 raft_consensus.cc:374] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:53.173918 14621 raft_consensus.cc:729] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9f8ac583194a2fb0210c759d54eb16, State: Initialized, Role: FOLLOWER
I20260806 23:51:53.174037 14623 tablet_bootstrap.cc:492] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: No bootstrap required, opened a new log
I20260806 23:51:53.174134 14623 ts_tablet_manager.cc:1397] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 23:51:53.174234 14621 consensus_queue.cc:260] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.174561 14621 ts_tablet_manager.cc:1428] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:53.174686 14622 tablet_bootstrap.cc:492] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: No bootstrap required, opened a new log
I20260806 23:51:53.174779 14622 ts_tablet_manager.cc:1397] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:51:53.175253 14622 raft_consensus.cc:348] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.175372 14622 raft_consensus.cc:374] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:53.175413 14622 raft_consensus.cc:729] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7b0f2de387401faa23d07c6692d8d7, State: Initialized, Role: FOLLOWER
I20260806 23:51:53.175370 14623 raft_consensus.cc:348] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.175490 14623 raft_consensus.cc:374] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:53.175520 14623 raft_consensus.cc:729] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b402b105ce84188a922e83526b84b72, State: Initialized, Role: FOLLOWER
I20260806 23:51:53.175544 14622 consensus_queue.cc:260] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.175721 14623 consensus_queue.cc:260] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.175949 14622 ts_tablet_manager.cc:1428] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:53.176123 14623 ts_tablet_manager.cc:1428] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 23:51:53.201319 14631 raft_consensus.cc:482] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:51:53.201505 14631 raft_consensus.cc:504] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.202672 14631 leader_election.cc:288] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129:34361), 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535)
I20260806 23:51:53.207631 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.203099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:39387 (local address 127.10.225.129:34361)
0806 23:51:53.203456 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:53.203467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:53.203901 (+ 434us) server_negotiation.cc:407] Connection header received
0806 23:51:53.204055 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:53.204058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:53.204108 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:53.204189 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:53.204467 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.205019 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.206424 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.206896 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.207038 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.207088 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:53.207391 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 23:51:53.207485 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":145,"threads_started":1}
I20260806 23:51:53.207696 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.203014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:34361 (local address 127.10.225.131:39387)
0806 23:51:53.203782 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:53.203795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:53.203928 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:53.204206 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:53.204209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:53.204227 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:53.204366 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:53.204373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.205130 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.205134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:53.206185 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:53.206203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.207076 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.207080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:53.207240 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.207277 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 23:51:53.207353 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 23:51:53.207461 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":200,"thread_start_us":131,"threads_started":1}
I20260806 23:51:53.207746 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.202999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:37535 (local address 127.10.225.131:37101)
0806 23:51:53.203594 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:53.203612 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:53.203791 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:53.204062 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:53.204065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:53.204082 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:51:53.204240 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:53.204247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.205085 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.205088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:53.206130 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:53.206151 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.207193 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.207198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:53.207329 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.207350 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 23:51:53.207429 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 23:51:53.207523 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":28,"thread_start_us":195,"threads_started":1}
I20260806 23:51:53.208335 13937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e928a72b5841f388cdb37c4b23501a" candidate_uuid: "9d7b0f2de387401faa23d07c6692d8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9f8ac583194a2fb0210c759d54eb16" is_pre_election: true
I20260806 23:51:53.208592 13937 raft_consensus.cc:2454] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7b0f2de387401faa23d07c6692d8d7 in term 0.
I20260806 23:51:53.208765 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.203236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:37101 (local address 127.10.225.130:37535)
0806 23:51:53.203660 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:53.203709 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:53.203728 (+ 19us) server_negotiation.cc:407] Connection header received
0806 23:51:53.203908 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:53.203911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:53.203960 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:53.204044 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:51:53.204344 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.204971 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.206366 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.206898 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.207043 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.207106 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:53.207502 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 23:51:53.207601 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":143,"threads_started":1}
I20260806 23:51:53.209046 14269 leader_election.cc:302] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7b0f2de387401faa23d07c6692d8d7, ba9f8ac583194a2fb0210c759d54eb16; no voters:
I20260806 23:51:53.209144 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e928a72b5841f388cdb37c4b23501a" candidate_uuid: "9d7b0f2de387401faa23d07c6692d8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72" is_pre_election: true
I20260806 23:51:53.209300 14156 raft_consensus.cc:2454] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7b0f2de387401faa23d07c6692d8d7 in term 0.
I20260806 23:51:53.209519 14631 raft_consensus.cc:2781] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:51:53.209589 14631 raft_consensus.cc:482] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:51:53.209630 14631 raft_consensus.cc:3037] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:53.210786 14631 raft_consensus.cc:504] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.211138 14631 leader_election.cc:288] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [CANDIDATE]: Term 1 election: Requested vote from peers ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129:34361), 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535)
I20260806 23:51:53.211519 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e928a72b5841f388cdb37c4b23501a" candidate_uuid: "9d7b0f2de387401faa23d07c6692d8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72"
I20260806 23:51:53.211537 13937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e928a72b5841f388cdb37c4b23501a" candidate_uuid: "9d7b0f2de387401faa23d07c6692d8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9f8ac583194a2fb0210c759d54eb16"
I20260806 23:51:53.211647 13937 raft_consensus.cc:3037] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:53.211647 14156 raft_consensus.cc:3037] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:53.212944 14156 raft_consensus.cc:2454] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7b0f2de387401faa23d07c6692d8d7 in term 1.
I20260806 23:51:53.212947 13937 raft_consensus.cc:2454] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7b0f2de387401faa23d07c6692d8d7 in term 1.
I20260806 23:51:53.213230 14266 leader_election.cc:302] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b402b105ce84188a922e83526b84b72, 9d7b0f2de387401faa23d07c6692d8d7; no voters:
I20260806 23:51:53.213456 14631 raft_consensus.cc:2781] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:53.213660 14631 raft_consensus.cc:686] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7b0f2de387401faa23d07c6692d8d7, State: Running, Role: LEADER
I20260806 23:51:53.213924 14631 consensus_queue.cc:237] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.217499 13683 catalog_manager.cc:5495] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131). New cstate: current_term: 1 leader_uuid: "9d7b0f2de387401faa23d07c6692d8d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } health_report { overall_health: HEALTHY } } }
I20260806 23:51:53.235080 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.230995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.10.225.131:38093)
0806 23:51:53.231281 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:53.231290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:53.231303 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:51:53.231383 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:53.231386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:53.231442 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:53.231550 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:53.231812 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.232380 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.233379 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.233646 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.233729 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.234115 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:53.234646 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:53.234758 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:53.234869 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 23:51:53.234927 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":111,"threads_started":1}
I20260806 23:51:53.247263 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.243736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:36691 (local address 127.10.225.129:34361)
0806 23:51:53.243875 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:53.243880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:53.244077 (+ 197us) server_negotiation.cc:407] Connection header received
0806 23:51:53.244201 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:53.244204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:53.244253 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:53.244334 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:53.244800 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.245325 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.246252 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.246504 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.246608 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.246877 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:53.246951 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:53.247008 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:53.247069 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 23:51:53.247120 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:51:53.247267 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.243465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:34361 (local address 127.10.225.131:36691)
0806 23:51:53.243914 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:53.243932 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:53.244094 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:53.244441 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:53.244444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:53.244454 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:51:53.244607 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:53.244612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.245423 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.245426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:53.246116 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:53.246125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.246618 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.246621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:53.246703 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.246759 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:51:53.247014 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:51:53.247018 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:51:53.247061 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 23:51:53.247131 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":114,"threads_started":1}
I20260806 23:51:53.254047 14156 raft_consensus.cc:1261] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Refusing update from remote peer 9d7b0f2de387401faa23d07c6692d8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:53.254285 13937 raft_consensus.cc:1261] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Refusing update from remote peer 9d7b0f2de387401faa23d07c6692d8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:53.254511 14639 consensus_queue.cc:1035] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:53.254766 14631 consensus_queue.cc:1035] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:53.274091 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.270785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:38093 (local address 127.10.225.129:48269)
0806 23:51:53.271096 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:53.271111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:53.271207 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:53.271457 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:53.271461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:53.271469 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:51:53.271581 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:53.271588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.272324 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.272327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:53.272955 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:53.272964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.273387 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.273390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:53.273480 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.273518 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:51:53.273793 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:51:53.273801 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 23:51:53.273864 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 23:51:53.273951 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":117,"threads_started":1}
I20260806 23:51:53.274089 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:53.271074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:48269 (local address 127.10.225.131:38093)
0806 23:51:53.271197 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:53.271202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:53.271216 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:51:53.271294 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:53.271296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:53.271351 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:53.271428 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:51:53.271703 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.272224 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.273066 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:53.273290 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:53.273390 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:53.273626 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:51:53.273707 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:51:53.273775 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:53.273879 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 23:51:53.273943 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:51:53.301991 13683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:53.302258 13683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:53.305425 13683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: Sending DeleteTablet for 3 replicas of tablet 77e928a72b5841f388cdb37c4b23501a
I20260806 23:51:53.306188 14126 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e928a72b5841f388cdb37c4b23501a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:53 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:50856
I20260806 23:51:53.306187 14346 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e928a72b5841f388cdb37c4b23501a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:53 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:38766
I20260806 23:51:53.306202 13898 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e928a72b5841f388cdb37c4b23501a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:53 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:57900
I20260806 23:51:53.306394 14607 tablet_replica.cc:330] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: stopping tablet replica
I20260806 23:51:53.306393 14608 tablet_replica.cc:330] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: stopping tablet replica
I20260806 23:51:53.306465 14606 tablet_replica.cc:330] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: stopping tablet replica
I20260806 23:51:53.306515 14607 raft_consensus.cc:2229] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:53.306515 14608 raft_consensus.cc:2229] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:51:53.306586 14606 raft_consensus.cc:2229] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:53.306643 14607 raft_consensus.cc:2258] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:53.306730 14606 raft_consensus.cc:2258] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:53.306880 14608 raft_consensus.cc:2258] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:53.307760 14608 ts_tablet_manager.cc:1905] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:53.307756 14607 ts_tablet_manager.cc:1905] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:53.307835 14606 ts_tablet_manager.cc:1905] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:53.310469 14607 ts_tablet_manager.cc:1918] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:51:53.310467 14608 ts_tablet_manager.cc:1918] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:51:53.310547 14607 log.cc:1196] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/77e928a72b5841f388cdb37c4b23501a
I20260806 23:51:53.310547 14608 log.cc:1196] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/77e928a72b5841f388cdb37c4b23501a
I20260806 23:51:53.310751 14606 ts_tablet_manager.cc:1918] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:51:53.310825 14606 log.cc:1196] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/77e928a72b5841f388cdb37c4b23501a
I20260806 23:51:53.310892 14608 ts_tablet_manager.cc:1939] T 77e928a72b5841f388cdb37c4b23501a P 9d7b0f2de387401faa23d07c6692d8d7: Deleting consensus metadata
I20260806 23:51:53.311079 14606 ts_tablet_manager.cc:1939] T 77e928a72b5841f388cdb37c4b23501a P ba9f8ac583194a2fb0210c759d54eb16: Deleting consensus metadata
I20260806 23:51:53.311105 14607 ts_tablet_manager.cc:1939] T 77e928a72b5841f388cdb37c4b23501a P 9b402b105ce84188a922e83526b84b72: Deleting consensus metadata
I20260806 23:51:53.312203 13656 catalog_manager.cc:4841] TS 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535): tablet 77e928a72b5841f388cdb37c4b23501a (table test-table [id=62ac6f828b7543578c1d18997b4a8683]) successfully deleted
I20260806 23:51:53.312203 13657 catalog_manager.cc:4841] TS 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093): tablet 77e928a72b5841f388cdb37c4b23501a (table test-table [id=62ac6f828b7543578c1d18997b4a8683]) successfully deleted
I20260806 23:51:53.312601 13659 catalog_manager.cc:4841] TS ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129:34361): tablet 77e928a72b5841f388cdb37c4b23501a (table test-table [id=62ac6f828b7543578c1d18997b4a8683]) successfully deleted
I20260806 23:51:53.312844 13683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:51:53.313299 13683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:51:53.319245 13898 tablet_service.cc:1467] Processing CreateTablet for tablet 474ba743ea2f403ea346af192decbe38 (DEFAULT_TABLE table=test-table [id=48d8e4e22dc44974ba5a67782aa7cd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:53.319461 14346 tablet_service.cc:1467] Processing CreateTablet for tablet 474ba743ea2f403ea346af192decbe38 (DEFAULT_TABLE table=test-table [id=48d8e4e22dc44974ba5a67782aa7cd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:53.319619 13898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474ba743ea2f403ea346af192decbe38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:53.319875 14346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474ba743ea2f403ea346af192decbe38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:53.320461 14126 tablet_service.cc:1467] Processing CreateTablet for tablet 474ba743ea2f403ea346af192decbe38 (DEFAULT_TABLE table=test-table [id=48d8e4e22dc44974ba5a67782aa7cd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:51:53.320771 14126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474ba743ea2f403ea346af192decbe38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:53.322743 14621 tablet_bootstrap.cc:492] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16: Bootstrap starting.
I20260806 23:51:53.323184 14623 tablet_bootstrap.cc:492] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: Bootstrap starting.
I20260806 23:51:53.323266 14622 tablet_bootstrap.cc:492] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: Bootstrap starting.
I20260806 23:51:53.323729 14621 tablet_bootstrap.cc:654] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:53.324174 14623 tablet_bootstrap.cc:654] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:53.324239 14622 tablet_bootstrap.cc:654] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:53.325073 14621 tablet_bootstrap.cc:492] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16: No bootstrap required, opened a new log
I20260806 23:51:53.325168 14621 ts_tablet_manager.cc:1397] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:51:53.325587 14621 raft_consensus.cc:348] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.325707 14621 raft_consensus.cc:374] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:53.325737 14621 raft_consensus.cc:729] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9f8ac583194a2fb0210c759d54eb16, State: Initialized, Role: FOLLOWER
I20260806 23:51:53.325860 14621 consensus_queue.cc:260] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.326118 14621 ts_tablet_manager.cc:1428] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:51:53.326275 14622 tablet_bootstrap.cc:492] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: No bootstrap required, opened a new log
I20260806 23:51:53.326362 14622 ts_tablet_manager.cc:1397] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 23:51:53.326771 14622 raft_consensus.cc:348] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.326891 14622 raft_consensus.cc:374] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:53.326927 14622 raft_consensus.cc:729] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7b0f2de387401faa23d07c6692d8d7, State: Initialized, Role: FOLLOWER
I20260806 23:51:53.327056 14622 consensus_queue.cc:260] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.327426 14622 ts_tablet_manager.cc:1428] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 23:51:53.328795 14623 tablet_bootstrap.cc:492] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: No bootstrap required, opened a new log
I20260806 23:51:53.328895 14623 ts_tablet_manager.cc:1397] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260806 23:51:53.329384 14623 raft_consensus.cc:348] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.329532 14623 raft_consensus.cc:374] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:53.329574 14623 raft_consensus.cc:729] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b402b105ce84188a922e83526b84b72, State: Initialized, Role: FOLLOWER
I20260806 23:51:53.329713 14623 consensus_queue.cc:260] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.330008 14623 ts_tablet_manager.cc:1428] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:51:53.374714 14652 raft_consensus.cc:482] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:51:53.374961 14652 raft_consensus.cc:504] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.375635 14652 leader_election.cc:288] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129:34361), 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535)
I20260806 23:51:53.376302 13937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474ba743ea2f403ea346af192decbe38" candidate_uuid: "9d7b0f2de387401faa23d07c6692d8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9f8ac583194a2fb0210c759d54eb16" is_pre_election: true
I20260806 23:51:53.376302 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474ba743ea2f403ea346af192decbe38" candidate_uuid: "9d7b0f2de387401faa23d07c6692d8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72" is_pre_election: true
I20260806 23:51:53.376556 14156 raft_consensus.cc:2454] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7b0f2de387401faa23d07c6692d8d7 in term 0.
I20260806 23:51:53.376612 13937 raft_consensus.cc:2454] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7b0f2de387401faa23d07c6692d8d7 in term 0.
I20260806 23:51:53.376916 14266 leader_election.cc:302] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b402b105ce84188a922e83526b84b72, 9d7b0f2de387401faa23d07c6692d8d7; no voters:
I20260806 23:51:53.377152 14652 raft_consensus.cc:2781] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:51:53.377221 14652 raft_consensus.cc:482] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:51:53.377260 14652 raft_consensus.cc:3037] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:53.378544 14652 raft_consensus.cc:504] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.378959 14652 leader_election.cc:288] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [CANDIDATE]: Term 1 election: Requested vote from peers ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129:34361), 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535)
I20260806 23:51:53.379359 14156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474ba743ea2f403ea346af192decbe38" candidate_uuid: "9d7b0f2de387401faa23d07c6692d8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b402b105ce84188a922e83526b84b72"
I20260806 23:51:53.379360 13937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474ba743ea2f403ea346af192decbe38" candidate_uuid: "9d7b0f2de387401faa23d07c6692d8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9f8ac583194a2fb0210c759d54eb16"
I20260806 23:51:53.379498 14156 raft_consensus.cc:3037] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:53.379498 13937 raft_consensus.cc:3037] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:53.380729 13937 raft_consensus.cc:2454] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7b0f2de387401faa23d07c6692d8d7 in term 1.
I20260806 23:51:53.380729 14156 raft_consensus.cc:2454] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7b0f2de387401faa23d07c6692d8d7 in term 1.
I20260806 23:51:53.381057 14266 leader_election.cc:302] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b402b105ce84188a922e83526b84b72, 9d7b0f2de387401faa23d07c6692d8d7; no voters:
I20260806 23:51:53.381261 14652 raft_consensus.cc:2781] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:53.381364 14652 raft_consensus.cc:686] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7b0f2de387401faa23d07c6692d8d7, State: Running, Role: LEADER
I20260806 23:51:53.381510 14652 consensus_queue.cc:237] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } }
I20260806 23:51:53.383301 13687 catalog_manager.cc:5495] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131). New cstate: current_term: 1 leader_uuid: "9d7b0f2de387401faa23d07c6692d8d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7b0f2de387401faa23d07c6692d8d7" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38093 } health_report { overall_health: HEALTHY } } }
I20260806 23:51:53.408725 14156 raft_consensus.cc:1261] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Refusing update from remote peer 9d7b0f2de387401faa23d07c6692d8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:53.408725 13937 raft_consensus.cc:1261] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16 [term 1 FOLLOWER]: Refusing update from remote peer 9d7b0f2de387401faa23d07c6692d8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:51:53.409121 14631 consensus_queue.cc:1035] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9f8ac583194a2fb0210c759d54eb16" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:51:53.409292 14652 consensus_queue.cc:1035] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b402b105ce84188a922e83526b84b72" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 23:51:53.430806 14570 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 23:51:54.989804 13687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:54.989975 13687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:51:54.992664 13687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b90a5f761c954125a3b67a716e8ac014: Sending DeleteTablet for 3 replicas of tablet 474ba743ea2f403ea346af192decbe38
I20260806 23:51:54.993451 13898 tablet_service.cc:1514] Processing DeleteTablet for tablet 474ba743ea2f403ea346af192decbe38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:54 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:57900
I20260806 23:51:54.993695 14126 tablet_service.cc:1514] Processing DeleteTablet for tablet 474ba743ea2f403ea346af192decbe38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:54 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:50856
I20260806 23:51:54.993700 14346 tablet_service.cc:1514] Processing DeleteTablet for tablet 474ba743ea2f403ea346af192decbe38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:51:54 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:38766
I20260806 23:51:54.994007 14697 tablet_replica.cc:330] T 474ba743ea2f403ea346af192decbe38 P ba9f8ac583194a2fb0210c759d54eb16: stopping tablet replica
I20260806 23:51:54.994036 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 13796
I20260806 23:51:55.020355 14698 tablet_replica.cc:330] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: stopping tablet replica
I20260806 23:51:55.020551 14698 raft_consensus.cc:2229] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:51:55.020821 14698 raft_consensus.cc:2258] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:55.022030 14698 ts_tablet_manager.cc:1905] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:55.025197 14698 ts_tablet_manager.cc:1918] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:51:55.025281 14698 log.cc:1196] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/474ba743ea2f403ea346af192decbe38
I20260806 23:51:55.025530 14698 ts_tablet_manager.cc:1939] T 474ba743ea2f403ea346af192decbe38 P 9d7b0f2de387401faa23d07c6692d8d7: Deleting consensus metadata
I20260806 23:51:55.026329 13657 catalog_manager.cc:4841] TS 9d7b0f2de387401faa23d07c6692d8d7 (127.10.225.131:38093): tablet 474ba743ea2f403ea346af192decbe38 (table test-table [id=48d8e4e22dc44974ba5a67782aa7cd01]) successfully deleted
I20260806 23:51:55.041393 14699 tablet_replica.cc:330] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: stopping tablet replica
I20260806 23:51:55.041638 14699 raft_consensus.cc:2229] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:51:55.041810 14699 raft_consensus.cc:2258] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:51:55.043262 14699 ts_tablet_manager.cc:1905] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:51:55.046655 14699 ts_tablet_manager.cc:1918] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 23:51:55.046763 14699 log.cc:1196] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/474ba743ea2f403ea346af192decbe38
I20260806 23:51:55.046998 14699 ts_tablet_manager.cc:1939] T 474ba743ea2f403ea346af192decbe38 P 9b402b105ce84188a922e83526b84b72: Deleting consensus metadata
I20260806 23:51:55.047828 13656 catalog_manager.cc:4841] TS 9b402b105ce84188a922e83526b84b72 (127.10.225.130:37535): tablet 474ba743ea2f403ea346af192decbe38 (table test-table [id=48d8e4e22dc44974ba5a67782aa7cd01]) successfully deleted
W20260806 23:51:55.049994 13659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:34361 (error 108)
I20260806 23:51:55.050199 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 14016
I20260806 23:51:55.056957 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:55.056446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:34361 (local address 127.0.0.1:45352)
0806 23:51:55.056736 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:55.056824 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:34361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
W20260806 23:51:55.057117 13659 catalog_manager.cc:4584] TS ba9f8ac583194a2fb0210c759d54eb16 (127.10.225.129:34361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 474ba743ea2f403ea346af192decbe38: Network error: Client connection negotiation failed: client connection to 127.10.225.129:34361: connect: Connection refused (error 111)
I20260806 23:51:55.062851 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 14241
I20260806 23:51:55.080297 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 13630
2026-08-06T23:51:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6806 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:51:55 dist-test-slave-nj21 krb5kdc[14707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:51:55 dist-test-slave-nj21 krb5kdc[14707](info): set up 2 sockets
Aug 06 23:51:55 dist-test-slave-nj21 krb5kdc[14707](info): commencing operation
krb5kdc: starting...
W20260806 23:51:56.292910 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:51:56 dist-test-slave-nj21 krb5kdc[14707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:51:56.438642 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38035
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:38035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:56.571946 14725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:56.572242 14725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:56.572283 14725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:56.576474 14725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:51:56.576545 14725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:56.576565 14725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:56.576579 14725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:51:56.576594 14725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:51:56.582206 14725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:38035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:56.583343 14725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:56.584439 14725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:56.593513 14737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:56.593459 14738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:56.593526 14741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:56.593794 14725 server_base.cc:1034] running on GCE node
I20260806 23:51:56.594396 14725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:56.595562 14725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:56.596762 14725 hybrid_clock.cc:648] HybridClock initialized: now 1786060316596743 us; error 40 us; skew 500 ppm
Aug 06 23:51:56 dist-test-slave-nj21 krb5kdc[14707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:56.599968 14725 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:51:56.601519 14725 webserver.cc:460] Webserver started at http://127.10.225.190:44279/ using document root <none> and password file <none>
I20260806 23:51:56.602326 14725 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:56.602411 14725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:56.602702 14725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:56.605654 14725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "5f7c3d69bccb4ad49585f576097199eb"
format_stamp: "Formatted at 2026-08-06 23:51:56 on dist-test-slave-nj21"
server_key: "23ad8b7631c9f9f2fe79440b1831df2b"
server_key_iv: "74adba38d377da30df52f7ad3c1cf5ea"
server_key_version: "encryptionkey@0"
I20260806 23:51:56.606424 14725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "5f7c3d69bccb4ad49585f576097199eb"
format_stamp: "Formatted at 2026-08-06 23:51:56 on dist-test-slave-nj21"
server_key: "23ad8b7631c9f9f2fe79440b1831df2b"
server_key_iv: "74adba38d377da30df52f7ad3c1cf5ea"
server_key_version: "encryptionkey@0"
I20260806 23:51:56.611788 14725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 23:51:56.615000 14755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:56.616899 14725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:51:56.617110 14725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "5f7c3d69bccb4ad49585f576097199eb"
format_stamp: "Formatted at 2026-08-06 23:51:56 on dist-test-slave-nj21"
server_key: "23ad8b7631c9f9f2fe79440b1831df2b"
server_key_iv: "74adba38d377da30df52f7ad3c1cf5ea"
server_key_version: "encryptionkey@0"
I20260806 23:51:56.617260 14725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:56.630259 14725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:56.633659 14725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:56.633860 14725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:56.641523 14725 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:38035
I20260806 23:51:56.641651 14843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:38035 every 8 connection(s)
I20260806 23:51:56.642637 14725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:51:56.646153 14844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:56.648502 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 14725
I20260806 23:51:56.648614 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:51:56.648883 11142 external_mini_cluster.cc:1428] Setting key 0987a15c1be3d3d8d4536e21321bf501
I20260806 23:51:56.653846 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb: Bootstrap starting.
I20260806 23:51:56.656677 14844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:56.657573 14844 log.cc:824] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:51:56 dist-test-slave-nj21 krb5kdc[14707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:56.660385 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb: No bootstrap required, opened a new log
I20260806 23:51:56.664203 14844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7c3d69bccb4ad49585f576097199eb" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38035 } }
I20260806 23:51:56.664546 14844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:56.664613 14844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7c3d69bccb4ad49585f576097199eb, State: Initialized, Role: FOLLOWER
I20260806 23:51:56.665093 14844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7c3d69bccb4ad49585f576097199eb" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38035 } }
I20260806 23:51:56.665234 14844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:51:56.665308 14844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:51:56.665400 14844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:56.665858 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:56.650452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.10.225.190:38035)
0806 23:51:56.650943 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:56.650960 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:56.650997 (+ 37us) server_negotiation.cc:407] Connection header received
0806 23:51:56.651707 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:56.651730 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:56.652085 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:56.652368 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:56.653359 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:56.654431 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:56.655586 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:56.656344 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:56.656545 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:56.659835 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:56.659843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:56.659867 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:56.659905 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:56.661783 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:56.662305 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:56.662311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:56.662318 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:56.662449 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:56.662769 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:56.662773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:56.662774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:56.663338 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:56.663705 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:56.664204 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 23:51:56.664568 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":155,"threads_started":1}
I20260806 23:51:56.666539 14844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7c3d69bccb4ad49585f576097199eb" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38035 } }
I20260806 23:51:56.666937 14844 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f7c3d69bccb4ad49585f576097199eb; no voters:
I20260806 23:51:56.667309 14844 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:51:56.667424 14851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:56.667697 14851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [term 1 LEADER]: Becoming Leader. State: Replica: 5f7c3d69bccb4ad49585f576097199eb, State: Running, Role: LEADER
I20260806 23:51:56.668346 14851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7c3d69bccb4ad49585f576097199eb" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38035 } }
I20260806 23:51:56.669205 14844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:51:56.670280 14851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7c3d69bccb4ad49585f576097199eb. Latest consensus state: current_term: 1 leader_uuid: "5f7c3d69bccb4ad49585f576097199eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7c3d69bccb4ad49585f576097199eb" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38035 } } }
I20260806 23:51:56.670437 14851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:56.671032 14860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:51:56.671859 14852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f7c3d69bccb4ad49585f576097199eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7c3d69bccb4ad49585f576097199eb" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38035 } } }
I20260806 23:51:56.672145 14852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:56.675174 14860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:51:56.682973 14860 catalog_manager.cc:1347] Generated new cluster ID: a1c04e956c994dce8103576d0f9de793
I20260806 23:51:56.683072 14860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:51:56.690958 14860 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:51:56.692157 14860 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:51:56.697882 14860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f7c3d69bccb4ad49585f576097199eb: Generated new TSK 0
I20260806 23:51:56.698944 14860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:51:56.734982 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38035
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:56.863577 14880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:56.863943 14880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:56.863992 14880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:56.868121 14880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:56.868191 14880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:56.868252 14880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:51:56.874033 14880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:56.875151 14880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:56.876394 14880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:56.886466 14897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:56.886487 14892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:56.886451 14893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:56.886899 14880 server_base.cc:1034] running on GCE node
I20260806 23:51:56.887328 14880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:56.888037 14880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:56.889227 14880 hybrid_clock.cc:648] HybridClock initialized: now 1786060316889208 us; error 29 us; skew 500 ppm
Aug 06 23:51:56 dist-test-slave-nj21 krb5kdc[14707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:56.892727 14880 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:51:56.893939 14880 webserver.cc:460] Webserver started at http://127.10.225.129:33083/ using document root <none> and password file <none>
I20260806 23:51:56.894582 14880 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:56.894652 14880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:56.894872 14880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:56.897122 14880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "5312a188fb134d739253b5aeeca327b7"
format_stamp: "Formatted at 2026-08-06 23:51:56 on dist-test-slave-nj21"
server_key: "1ef0a5d48d99cae5a4d10c409b63df62"
server_key_iv: "bb59e71356e71cd518b8867e420bdf4f"
server_key_version: "encryptionkey@0"
I20260806 23:51:56.897699 14880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "5312a188fb134d739253b5aeeca327b7"
format_stamp: "Formatted at 2026-08-06 23:51:56 on dist-test-slave-nj21"
server_key: "1ef0a5d48d99cae5a4d10c409b63df62"
server_key_iv: "bb59e71356e71cd518b8867e420bdf4f"
server_key_version: "encryptionkey@0"
I20260806 23:51:56.901829 14880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:51:56.904656 14915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:56.906838 14880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:51:56.906973 14880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "5312a188fb134d739253b5aeeca327b7"
format_stamp: "Formatted at 2026-08-06 23:51:56 on dist-test-slave-nj21"
server_key: "1ef0a5d48d99cae5a4d10c409b63df62"
server_key_iv: "bb59e71356e71cd518b8867e420bdf4f"
server_key_version: "encryptionkey@0"
I20260806 23:51:56.907099 14880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:56.922901 14880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:56.926443 14880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:56.926630 14880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:56.927299 14880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:51:56.928406 14880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:56.928465 14880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:56.928514 14880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:56.928537 14880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:56.939764 14880 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:36715
I20260806 23:51:56.939905 15106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:36715 every 8 connection(s)
I20260806 23:51:56.940864 14880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:51:56.943385 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 14880
I20260806 23:51:56.943544 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:51:56.943917 11142 external_mini_cluster.cc:1428] Setting key 34da8ffea7b3e0cf8efb266ab149f548
Aug 06 23:51:56 dist-test-slave-nj21 krb5kdc[14707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060316, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:51:56.959906 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:56.943600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:35793 (local address 127.10.225.190:38035)
0806 23:51:56.943840 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:56.943845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:56.945251 (+ 1406us) server_negotiation.cc:407] Connection header received
0806 23:51:56.947091 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:56.947094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:56.947150 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:56.947226 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:56.948889 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:56.949691 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:56.951265 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:56.951640 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:56.951783 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:56.954957 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:56.954962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:56.954965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:56.955001 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:56.956585 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:56.957234 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:56.957237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:56.957239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:56.957471 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:56.958214 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:56.958218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:56.958220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:56.958603 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:56.958755 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:56.959484 (+ 729us) server_negotiation.cc:299] Negotiation successful
0806 23:51:56.959699 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 23:51:56.961133 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:56.944221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38035 (local address 127.10.225.129:35793)
0806 23:51:56.945041 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:56.945105 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:56.946765 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:56.947466 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:56.947476 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:56.948049 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:56.948690 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:56.948705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:56.949904 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:56.949908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:56.951060 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:56.951078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:56.951884 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:56.951889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:56.952088 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:56.952728 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:56.952745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:56.954706 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:56.956733 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:56.956740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:56.956759 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:56.957082 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:56.957797 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:56.957802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:56.957804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:56.958087 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:56.958786 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:56.958794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:56.959140 (+ 346us) client_negotiation.cc:769] Sending connection context
0806 23:51:56.959422 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 23:51:56.959946 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":212,"threads_started":1}
I20260806 23:51:56.962662 15107 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38035
I20260806 23:51:56.962984 15107 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:56.963806 15107 heartbeater.cc:511] Master 127.10.225.190:38035 requested a full tablet report, sending...
I20260806 23:51:56.966027 14783 ts_manager.cc:194] Registered new tserver with Master: 5312a188fb134d739253b5aeeca327b7 (127.10.225.129:36715)
I20260806 23:51:56.967267 14783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:35793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:51:56.979146 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38035
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:57.112056 15114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:57.112414 15114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:57.112486 15114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:57.117264 15114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:57.117344 15114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:57.117436 15114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:51:57.123526 15114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:57.124748 15114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:57.126044 15114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:57.136632 15125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:57.136673 15129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:57.136974 15124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:57.136947 15114 server_base.cc:1034] running on GCE node
I20260806 23:51:57.137765 15114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:57.138696 15114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:57.139909 15114 hybrid_clock.cc:648] HybridClock initialized: now 1786060317139880 us; error 37 us; skew 500 ppm
Aug 06 23:51:57 dist-test-slave-nj21 krb5kdc[14707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:57.144001 15114 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:51:57.145329 15114 webserver.cc:460] Webserver started at http://127.10.225.130:34219/ using document root <none> and password file <none>
I20260806 23:51:57.146055 15114 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:57.146129 15114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:57.146401 15114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:57.149075 15114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "fa781de099c941518caa6ed8a45afeca"
format_stamp: "Formatted at 2026-08-06 23:51:57 on dist-test-slave-nj21"
server_key: "827410656921f9c88fb513d630604346"
server_key_iv: "0ff86eb0503b3082e302f1383dad3afd"
server_key_version: "encryptionkey@0"
I20260806 23:51:57.149775 15114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "fa781de099c941518caa6ed8a45afeca"
format_stamp: "Formatted at 2026-08-06 23:51:57 on dist-test-slave-nj21"
server_key: "827410656921f9c88fb513d630604346"
server_key_iv: "0ff86eb0503b3082e302f1383dad3afd"
server_key_version: "encryptionkey@0"
I20260806 23:51:57.155229 15114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 23:51:57.158725 15147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:57.160537 15114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:51:57.160687 15114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "fa781de099c941518caa6ed8a45afeca"
format_stamp: "Formatted at 2026-08-06 23:51:57 on dist-test-slave-nj21"
server_key: "827410656921f9c88fb513d630604346"
server_key_iv: "0ff86eb0503b3082e302f1383dad3afd"
server_key_version: "encryptionkey@0"
I20260806 23:51:57.160843 15114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:57.175311 15114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:57.180038 15114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:57.180260 15114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:57.181037 15114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:51:57.182098 15114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:57.182150 15114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:57.182196 15114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:57.182214 15114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:57.192950 15114 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:46589
I20260806 23:51:57.192960 15346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:46589 every 8 connection(s)
I20260806 23:51:57.193976 15114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:51:57.198721 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 15114
I20260806 23:51:57.198866 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:51:57.199154 11142 external_mini_cluster.cc:1428] Setting key a85e3a4f430bd3e2a59f39fc1a4a696c
Aug 06 23:51:57 dist-test-slave-nj21 krb5kdc[14707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:57.210282 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:57.196326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:50577 (local address 127.10.225.190:38035)
0806 23:51:57.196488 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:57.196492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:57.197488 (+ 996us) server_negotiation.cc:407] Connection header received
0806 23:51:57.199051 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:57.199055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:57.199135 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:57.199268 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:57.200693 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:57.201326 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:57.202948 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:57.203296 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:57.203431 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:57.205968 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:57.205972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:57.205976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:57.206008 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:57.207816 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:57.208431 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:57.208434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:57.208436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:57.208508 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:57.208885 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:57.208888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:57.208889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:57.209144 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:57.209225 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:57.209970 (+ 745us) server_negotiation.cc:299] Negotiation successful
0806 23:51:57.210108 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:51:57.211328 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:57.196723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38035 (local address 127.10.225.130:50577)
0806 23:51:57.197245 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:57.197372 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:57.198696 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:57.199481 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:57.199490 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:57.199993 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:57.200518 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:57.200532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:57.201484 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:57.201488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:57.202778 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:57.202790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:57.203457 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:57.203460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:57.203607 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:57.204206 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:57.204224 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:57.205766 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:57.207935 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:57.207943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:57.207962 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:57.208339 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:57.208588 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:57.208591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:57.208592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:57.208779 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:57.209258 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:57.209266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:57.209643 (+ 377us) client_negotiation.cc:769] Sending connection context
0806 23:51:57.209889 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 23:51:57.210292 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":134,"threads_started":1}
I20260806 23:51:57.212694 15348 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38035
I20260806 23:51:57.212965 15348 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
I20260806 23:51:57.213534 15348 heartbeater.cc:511] Master 127.10.225.190:38035 requested a full tablet report, sending...
I20260806 23:51:57.214809 14783 ts_manager.cc:194] Registered new tserver with Master: fa781de099c941518caa6ed8a45afeca (127.10.225.130:46589)
I20260806 23:51:57.215420 14783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:50577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:51:57.230921 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38035
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:57.358809 15358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:57.359117 15358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:57.359165 15358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:57.363337 15358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:57.363417 15358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:57.363514 15358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:51:57.369217 15358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:38035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:57.370450 15358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:57.371582 15358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:57.381690 15370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:57.381682 15367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:57.381836 15358 server_base.cc:1034] running on GCE node
W20260806 23:51:57.381765 15375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:57.382519 15358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:57.383222 15358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:57.384397 15358 hybrid_clock.cc:648] HybridClock initialized: now 1786060317384378 us; error 25 us; skew 500 ppm
Aug 06 23:51:57 dist-test-slave-nj21 krb5kdc[14707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:57.388196 15358 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:51:57.389266 15358 webserver.cc:460] Webserver started at http://127.10.225.131:39579/ using document root <none> and password file <none>
I20260806 23:51:57.389899 15358 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:57.389971 15358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:57.390175 15358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:57.392470 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "e1fd7dee81034b3cb28f2c83acbff7be"
format_stamp: "Formatted at 2026-08-06 23:51:57 on dist-test-slave-nj21"
server_key: "47b72392ef0a6bb17816a7170b029d53"
server_key_iv: "c50ac08d1e77dc109f089c3bd1f15d97"
server_key_version: "encryptionkey@0"
I20260806 23:51:57.393039 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "e1fd7dee81034b3cb28f2c83acbff7be"
format_stamp: "Formatted at 2026-08-06 23:51:57 on dist-test-slave-nj21"
server_key: "47b72392ef0a6bb17816a7170b029d53"
server_key_iv: "c50ac08d1e77dc109f089c3bd1f15d97"
server_key_version: "encryptionkey@0"
I20260806 23:51:57.397449 15358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:51:57.400691 15387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:57.402745 15358 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:51:57.402871 15358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "e1fd7dee81034b3cb28f2c83acbff7be"
format_stamp: "Formatted at 2026-08-06 23:51:57 on dist-test-slave-nj21"
server_key: "47b72392ef0a6bb17816a7170b029d53"
server_key_iv: "c50ac08d1e77dc109f089c3bd1f15d97"
server_key_version: "encryptionkey@0"
I20260806 23:51:57.403000 15358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:57.427554 15358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:57.431597 15358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:57.431821 15358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:57.432461 15358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:51:57.433372 15358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:51:57.433427 15358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:57.433470 15358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:51:57.433490 15358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:57.444353 15565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:36129 every 8 connection(s)
I20260806 23:51:57.444401 15358 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:36129
I20260806 23:51:57.445794 15358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:51:57.449795 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 15358
I20260806 23:51:57.449923 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:51:57.450198 11142 external_mini_cluster.cc:1428] Setting key 6d9d09b8c520419b523c8d3d2128b779
Aug 06 23:51:57 dist-test-slave-nj21 krb5kdc[14707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060317, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:57.462778 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:57.448425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:37851 (local address 127.10.225.190:38035)
0806 23:51:57.448560 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:57.448564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:57.449701 (+ 1137us) server_negotiation.cc:407] Connection header received
0806 23:51:57.450711 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:57.450715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:57.450784 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:57.450869 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:57.452493 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:57.453196 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:57.454637 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:57.454993 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:57.455093 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:57.458434 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:57.458439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:57.458441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:57.458476 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:57.460175 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:57.460807 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:57.460811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:57.460812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:57.460925 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:57.461435 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:57.461438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:57.461439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:57.461657 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:57.461738 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:57.462511 (+ 773us) server_negotiation.cc:299] Negotiation successful
0806 23:51:57.462640 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:51:57.464147 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:57.448788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38035 (local address 127.10.225.131:37851)
0806 23:51:57.449543 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 23:51:57.449587 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:51:57.450452 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:51:57.451069 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:51:57.451079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:51:57.451567 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:51:57.452295 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:57.452317 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:57.453358 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:57.453361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:57.454482 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:51:57.454495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:51:57.455224 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:51:57.455229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:51:57.455450 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:57.456351 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:51:57.456371 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:51:57.458184 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:51:57.460323 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:57.460329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:57.460347 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:57.460668 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:57.461075 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:51:57.461079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:51:57.461081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:51:57.461340 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:51:57.461800 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:51:57.461810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:51:57.462190 (+ 380us) client_negotiation.cc:769] Sending connection context
0806 23:51:57.462410 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 23:51:57.462791 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":134,"threads_started":1}
I20260806 23:51:57.465919 15567 heartbeater.cc:347] Connected to a master server at 127.10.225.190:38035
I20260806 23:51:57.466231 15567 heartbeater.cc:464] Registering TS with master...
I20260806 23:51:57.466957 15567 heartbeater.cc:511] Master 127.10.225.190:38035 requested a full tablet report, sending...
I20260806 23:51:57.468331 14783 ts_manager.cc:194] Registered new tserver with Master: e1fd7dee81034b3cb28f2c83acbff7be (127.10.225.131:36129)
I20260806 23:51:57.468910 14783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:37851
I20260806 23:51:57.478080 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 23:51:57.494887 14849 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:51:57.492984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.10.225.190:38035)
0806 23:51:57.493126 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:57.493130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:57.493409 (+ 279us) server_negotiation.cc:407] Connection header received
0806 23:51:57.493498 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:57.493500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:57.493548 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:57.493622 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:57.494727 (+ 1105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60504: BlockingRecv error: recv got EOF from 127.0.0.1:60504 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:51:57.495972 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 14880
I20260806 23:51:57.507966 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 15114
I20260806 23:51:57.530652 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 15358
I20260806 23:51:57.542847 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 14725
2026-08-06T23:51:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2464 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:51:57 dist-test-slave-nj21 krb5kdc[15583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:51:57 dist-test-slave-nj21 krb5kdc[15583](info): set up 2 sockets
Aug 06 23:51:57 dist-test-slave-nj21 krb5kdc[15583](info): commencing operation
krb5kdc: starting...
W20260806 23:51:58.708050 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:51:58 dist-test-slave-nj21 krb5kdc[15583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:51:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:51:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:51:58.851301 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41837
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:41837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:51:58.973627 15608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:51:58.973919 15608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:51:58.973978 15608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:51:58.978227 15608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:51:58.978299 15608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:51:58.978327 15608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:51:58.978350 15608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:51:58.978375 15608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:51:58.983812 15608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:41837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:51:58.984965 15608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:51:58.986022 15608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:51:58.995237 15619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:58.995226 15615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:51:58.995236 15614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:51:58.995746 15608 server_base.cc:1034] running on GCE node
I20260806 23:51:58.996665 15608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:51:58.998011 15608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:51:58.999197 15608 hybrid_clock.cc:648] HybridClock initialized: now 1786060318999168 us; error 44 us; skew 500 ppm
Aug 06 23:51:58 dist-test-slave-nj21 krb5kdc[15583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060318, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:51:59.002725 15608 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:51:59.004076 15608 webserver.cc:460] Webserver started at http://127.10.225.190:37373/ using document root <none> and password file <none>
I20260806 23:51:59.004804 15608 fs_manager.cc:359] Metadata directory not provided
I20260806 23:51:59.004877 15608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:51:59.005119 15608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:51:59.007575 15608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "5bc95cd0aa0d4a89a9829d33611dfb98"
format_stamp: "Formatted at 2026-08-06 23:51:59 on dist-test-slave-nj21"
server_key: "36aa414b1d71b8989fe8931827a2f690"
server_key_iv: "96b769892204b42978b501646834ccf4"
server_key_version: "encryptionkey@0"
I20260806 23:51:59.008214 15608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "5bc95cd0aa0d4a89a9829d33611dfb98"
format_stamp: "Formatted at 2026-08-06 23:51:59 on dist-test-slave-nj21"
server_key: "36aa414b1d71b8989fe8931827a2f690"
server_key_iv: "96b769892204b42978b501646834ccf4"
server_key_version: "encryptionkey@0"
I20260806 23:51:59.012516 15608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:51:59.015185 15629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:51:59.016610 15608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:51:59.016737 15608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "5bc95cd0aa0d4a89a9829d33611dfb98"
format_stamp: "Formatted at 2026-08-06 23:51:59 on dist-test-slave-nj21"
server_key: "36aa414b1d71b8989fe8931827a2f690"
server_key_iv: "96b769892204b42978b501646834ccf4"
server_key_version: "encryptionkey@0"
I20260806 23:51:59.016842 15608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:51:59.033700 15608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:51:59.037125 15608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:51:59.037312 15608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:51:59.045047 15608 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:41837
I20260806 23:51:59.045531 15703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:41837 every 8 connection(s)
I20260806 23:51:59.046116 15608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:51:59.049077 15708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:51:59.049831 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 15608
I20260806 23:51:59.049960 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:51:59.050262 11142 external_mini_cluster.cc:1428] Setting key 1c806b61375b92b2b5c2b9320d88dcba
I20260806 23:51:59.055994 15708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98: Bootstrap starting.
I20260806 23:51:59.058686 15708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98: Neither blocks nor log segments found. Creating new log.
I20260806 23:51:59.059713 15708 log.cc:824] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:51:59 dist-test-slave-nj21 krb5kdc[15583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:51:59.062201 15708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98: No bootstrap required, opened a new log
I20260806 23:51:59.066730 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:59.051995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.10.225.190:41837)
0806 23:51:59.052421 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:59.052437 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:59.052472 (+ 35us) server_negotiation.cc:407] Connection header received
0806 23:51:59.053150 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:59.053168 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:59.053510 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:59.053789 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:59.054762 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:59.055790 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:59.056985 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:59.057725 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:59.057931 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:59.061152 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:59.061161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:59.061180 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:51:59.061216 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:59.063181 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:59.063632 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:59.063638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:59.063644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:59.063829 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:51:59.064121 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:51:59.064124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:51:59.064126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:51:59.064717 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:59.064924 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:59.065193 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 23:51:59.065523 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":115,"threads_started":1}
I20260806 23:51:59.067164 15708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc95cd0aa0d4a89a9829d33611dfb98" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41837 } }
I20260806 23:51:59.067464 15708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:51:59.067550 15708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc95cd0aa0d4a89a9829d33611dfb98, State: Initialized, Role: FOLLOWER
I20260806 23:51:59.068056 15708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc95cd0aa0d4a89a9829d33611dfb98" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41837 } }
I20260806 23:51:59.068188 15708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:51:59.068244 15708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:51:59.068348 15708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:51:59.070205 15708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc95cd0aa0d4a89a9829d33611dfb98" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41837 } }
I20260806 23:51:59.070554 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:51:59.070809 15708 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc95cd0aa0d4a89a9829d33611dfb98; no voters:
I20260806 23:51:59.071345 15708 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:51:59.071434 15715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:51:59.071703 15715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [term 1 LEADER]: Becoming Leader. State: Replica: 5bc95cd0aa0d4a89a9829d33611dfb98, State: Running, Role: LEADER
I20260806 23:51:59.072101 15715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc95cd0aa0d4a89a9829d33611dfb98" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41837 } }
I20260806 23:51:59.073472 15708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:51:59.073841 15718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc95cd0aa0d4a89a9829d33611dfb98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc95cd0aa0d4a89a9829d33611dfb98" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41837 } } }
I20260806 23:51:59.073977 15718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:59.074203 15718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc95cd0aa0d4a89a9829d33611dfb98. Latest consensus state: current_term: 1 leader_uuid: "5bc95cd0aa0d4a89a9829d33611dfb98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc95cd0aa0d4a89a9829d33611dfb98" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41837 } } }
I20260806 23:51:59.074280 15718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98 [sys.catalog]: This master's current role is: LEADER
I20260806 23:51:59.075321 15725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:51:59.078863 15725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:51:59.084760 15725 catalog_manager.cc:1347] Generated new cluster ID: 9a3f8e5f99454938a96cc6c3808c2246
I20260806 23:51:59.084909 15725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:51:59.095744 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:51:59.089670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.10.225.190:41837)
0806 23:51:59.089840 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 23:51:59.089845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:51:59.089943 (+ 98us) server_negotiation.cc:407] Connection header received
0806 23:51:59.090086 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:51:59.090090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:51:59.090167 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:51:59.090309 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:51:59.091015 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:59.091937 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:59.093072 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:51:59.093473 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:51:59.093595 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:51:59.094526 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:51:59.094531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:51:59.094535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:51:59.095183 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:51:59.095235 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:51:59.095260 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:51:59.095364 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 23:51:59.095447 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 23:51:59.095521 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 23:51:59.106510 15725 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:51:59.107936 15725 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:51:59.120179 15725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc95cd0aa0d4a89a9829d33611dfb98: Generated new TSK 0
I20260806 23:51:59.121361 15725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:51:59.129106 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 15608
2026-08-06T23:51:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1583 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:51:59 dist-test-slave-nj21 krb5kdc[15758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:51:59 dist-test-slave-nj21 krb5kdc[15758](info): set up 2 sockets
Aug 06 23:51:59 dist-test-slave-nj21 krb5kdc[15758](info): commencing operation
krb5kdc: starting...
W20260806 23:52:00.299001 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:00 dist-test-slave-nj21 krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:52:00.447961 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34197
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:34197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:00.578415 15774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:00.578948 15774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:00.579016 15774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:00.586753 15774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:00.586871 15774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:00.586906 15774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:00.586932 15774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:00.586971 15774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:00.596104 15774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:34197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:00.598321 15774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:00.600198 15774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:00.609259 15788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:00.609298 15786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:00.609259 15785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:00.609819 15774 server_base.cc:1034] running on GCE node
I20260806 23:52:00.610419 15774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:00.611583 15774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:00.612815 15774 hybrid_clock.cc:648] HybridClock initialized: now 1786060320612787 us; error 36 us; skew 500 ppm
Aug 06 23:52:00 dist-test-slave-nj21 krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060320, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:00.621240 15774 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:52:00.622444 15774 webserver.cc:460] Webserver started at http://127.10.225.190:36685/ using document root <none> and password file <none>
I20260806 23:52:00.623068 15774 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:00.623138 15774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:00.623368 15774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:00.625608 15774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "eee00875f14d420a99e46ffc72e0536f"
format_stamp: "Formatted at 2026-08-06 23:52:00 on dist-test-slave-nj21"
server_key: "51ea096176ef47faf2d8bdff41409ab1"
server_key_iv: "66646007575799f371a86345ec3bef83"
server_key_version: "encryptionkey@0"
I20260806 23:52:00.626186 15774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "eee00875f14d420a99e46ffc72e0536f"
format_stamp: "Formatted at 2026-08-06 23:52:00 on dist-test-slave-nj21"
server_key: "51ea096176ef47faf2d8bdff41409ab1"
server_key_iv: "66646007575799f371a86345ec3bef83"
server_key_version: "encryptionkey@0"
I20260806 23:52:00.630448 15774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:52:00.633260 15801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:00.634613 15774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:52:00.634748 15774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "eee00875f14d420a99e46ffc72e0536f"
format_stamp: "Formatted at 2026-08-06 23:52:00 on dist-test-slave-nj21"
server_key: "51ea096176ef47faf2d8bdff41409ab1"
server_key_iv: "66646007575799f371a86345ec3bef83"
server_key_version: "encryptionkey@0"
I20260806 23:52:00.634863 15774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:00.647496 15774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:00.651093 15774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:00.651298 15774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:00.659044 15774 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:34197
I20260806 23:52:00.659322 15893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:34197 every 8 connection(s)
I20260806 23:52:00.660120 15774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:00.663750 15895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:00.666760 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 15774
I20260806 23:52:00.666903 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:00.667162 11142 external_mini_cluster.cc:1428] Setting key 7bc0234b5cc56dd0d8f297d56b6ab09b
I20260806 23:52:00.669994 15895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f: Bootstrap starting.
I20260806 23:52:00.673668 15895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:00.674760 15895 log.cc:824] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:52:00 dist-test-slave-nj21 krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:00.677989 15895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f: No bootstrap required, opened a new log
I20260806 23:52:00.683090 15895 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee00875f14d420a99e46ffc72e0536f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34197 } }
I20260806 23:52:00.683410 15895 raft_consensus.cc:374] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:00.683486 15895 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee00875f14d420a99e46ffc72e0536f, State: Initialized, Role: FOLLOWER
I20260806 23:52:00.684008 15895 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee00875f14d420a99e46ffc72e0536f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34197 } }
I20260806 23:52:00.684147 15895 raft_consensus.cc:388] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:00.684217 15895 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:00.684314 15895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:00.684463 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:00.668553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.10.225.190:34197)
0806 23:52:00.668996 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:00.669012 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:00.669048 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:52:00.670026 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:00.670083 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:00.670771 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:00.671085 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:00.672601 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:00.673764 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:00.674710 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:00.675298 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:00.675457 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:00.678855 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:00.678860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:00.678878 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:00.678911 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:00.680723 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:00.681496 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:00.681502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:00.681507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:00.681646 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:00.681963 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:00.681965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:00.681966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:00.682569 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:00.682744 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:00.683075 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 23:52:00.683404 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260806 23:52:00.685715 15895 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee00875f14d420a99e46ffc72e0536f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34197 } }
I20260806 23:52:00.686072 15895 leader_election.cc:302] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eee00875f14d420a99e46ffc72e0536f; no voters:
I20260806 23:52:00.686415 15895 leader_election.cc:288] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:00.686604 15901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:00.686955 15901 raft_consensus.cc:686] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [term 1 LEADER]: Becoming Leader. State: Replica: eee00875f14d420a99e46ffc72e0536f, State: Running, Role: LEADER
I20260806 23:52:00.687441 15901 consensus_queue.cc:237] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee00875f14d420a99e46ffc72e0536f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34197 } }
I20260806 23:52:00.687875 15895 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:00.689478 15903 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee00875f14d420a99e46ffc72e0536f. Latest consensus state: current_term: 1 leader_uuid: "eee00875f14d420a99e46ffc72e0536f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee00875f14d420a99e46ffc72e0536f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34197 } } }
I20260806 23:52:00.689738 15903 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:00.689949 15902 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eee00875f14d420a99e46ffc72e0536f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee00875f14d420a99e46ffc72e0536f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34197 } } }
I20260806 23:52:00.690060 15902 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f [sys.catalog]: This master's current role is: LEADER
W20260806 23:52:00.693795 15931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:52:00.693892 15931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:52:00.694008 15926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:00.695115 15926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:00.703485 15926 catalog_manager.cc:1347] Generated new cluster ID: b09744e165ac45fb8a8ac3f099414e93
I20260806 23:52:00.703580 15926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:00.719437 15926 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:00.720486 15926 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:00.730799 15926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eee00875f14d420a99e46ffc72e0536f: Generated new TSK 0
I20260806 23:52:00.731648 15926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:52:00.774608 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34197
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:00.921173 15942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:00.921473 15942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:00.921536 15942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:00.925643 15942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:00.925715 15942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:00.925799 15942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:00.931509 15942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:00.932760 15942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:00.934355 15942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:00.946254 15949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:00.946322 15956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:00.946317 15952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:00.946592 15942 server_base.cc:1034] running on GCE node
I20260806 23:52:00.946991 15942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:00.947755 15942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:00.948956 15942 hybrid_clock.cc:648] HybridClock initialized: now 1786060320948929 us; error 30 us; skew 500 ppm
Aug 06 23:52:00 dist-test-slave-nj21 krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060320, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:00.952014 15942 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:52:00.953305 15942 webserver.cc:460] Webserver started at http://127.10.225.129:33059/ using document root <none> and password file <none>
I20260806 23:52:00.953894 15942 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:00.953958 15942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:00.954133 15942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:00.956336 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "46451862fcb346a399df0e23ddc1398f"
format_stamp: "Formatted at 2026-08-06 23:52:00 on dist-test-slave-nj21"
server_key: "3b877f18c9f66db538e6cac69bee1fb0"
server_key_iv: "400cc481ca662cdf08e6f1a14f9fed3b"
server_key_version: "encryptionkey@0"
I20260806 23:52:00.956903 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "46451862fcb346a399df0e23ddc1398f"
format_stamp: "Formatted at 2026-08-06 23:52:00 on dist-test-slave-nj21"
server_key: "3b877f18c9f66db538e6cac69bee1fb0"
server_key_iv: "400cc481ca662cdf08e6f1a14f9fed3b"
server_key_version: "encryptionkey@0"
I20260806 23:52:00.960870 15942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:52:00.963729 15963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:00.965790 15942 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:52:00.965975 15942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "46451862fcb346a399df0e23ddc1398f"
format_stamp: "Formatted at 2026-08-06 23:52:00 on dist-test-slave-nj21"
server_key: "3b877f18c9f66db538e6cac69bee1fb0"
server_key_iv: "400cc481ca662cdf08e6f1a14f9fed3b"
server_key_version: "encryptionkey@0"
I20260806 23:52:00.966127 15942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:00.979125 15942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:00.982363 15942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:00.982537 15942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:00.983127 15942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:00.984017 15942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:00.984071 15942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:00.984112 15942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:00.984126 15942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:00.993191 15942 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:40463
I20260806 23:52:00.993211 16154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:40463 every 8 connection(s)
I20260806 23:52:00.994218 15942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:00.995599 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 15942
I20260806 23:52:00.995766 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:00.996104 11142 external_mini_cluster.cc:1428] Setting key 11ad5532e3dc479f12cce0ecb1c4359a
Aug 06 23:52:01 dist-test-slave-nj21 krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060320, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:52:01.011431 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:00.996182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:42321 (local address 127.10.225.190:34197)
0806 23:52:00.996314 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:00.996318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:00.997584 (+ 1266us) server_negotiation.cc:407] Connection header received
0806 23:52:00.999280 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:00.999283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:00.999340 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:00.999461 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:01.001371 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.001921 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.003755 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.004002 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.004121 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:01.007415 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:01.007420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:01.007423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:01.007456 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:01.009302 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:01.009829 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:01.009831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:01.009833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:01.009915 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:01.010262 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:01.010264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:01.010265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:01.010461 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:01.010538 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:01.011131 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 23:52:01.011285 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:52:01.012487 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:00.996651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34197 (local address 127.10.225.129:42321)
0806 23:52:00.997380 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:00.997455 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:00.999020 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:00.999721 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:00.999730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:01.000312 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:01.001029 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:01.001137 (+ 108us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.002080 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.002084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:01.003560 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:01.003574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.004224 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.004229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:01.004477 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:01.005340 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:01.005365 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:01.007201 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:01.009443 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:01.009449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:01.009474 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:01.009747 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:01.010007 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:01.010009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:01.010011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:01.010181 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:01.010575 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:01.010581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:01.010866 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 23:52:01.011056 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 23:52:01.011446 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":124,"threads_started":1}
I20260806 23:52:01.013863 16155 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34197
I20260806 23:52:01.014137 16155 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:01.014773 16155 heartbeater.cc:511] Master 127.10.225.190:34197 requested a full tablet report, sending...
I20260806 23:52:01.016637 15835 ts_manager.cc:194] Registered new tserver with Master: 46451862fcb346a399df0e23ddc1398f (127.10.225.129:40463)
I20260806 23:52:01.018775 15835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:42321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:52:01.035113 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34197
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:01.164937 16160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:01.165236 16160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:01.165292 16160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:01.169605 16160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:01.169695 16160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:01.169770 16160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:01.175571 16160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:01.177416 16160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:01.178855 16160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:01.191717 16170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:01.191722 16169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:01.191812 16175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:01.191953 16160 server_base.cc:1034] running on GCE node
I20260806 23:52:01.192503 16160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:01.193231 16160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:01.194445 16160 hybrid_clock.cc:648] HybridClock initialized: now 1786060321194414 us; error 36 us; skew 500 ppm
Aug 06 23:52:01 dist-test-slave-nj21 krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:01.198244 16160 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:52:01.199924 16160 webserver.cc:460] Webserver started at http://127.10.225.130:36535/ using document root <none> and password file <none>
I20260806 23:52:01.200706 16160 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:01.200783 16160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:01.201045 16160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:01.203749 16160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "1d9540cd8ae74432871fae5fa25bbdb6"
format_stamp: "Formatted at 2026-08-06 23:52:01 on dist-test-slave-nj21"
server_key: "ff75cf1a02e2e7aa63667c7da14e3dc1"
server_key_iv: "3f4712899b94670162b4e08205c6722c"
server_key_version: "encryptionkey@0"
I20260806 23:52:01.204466 16160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "1d9540cd8ae74432871fae5fa25bbdb6"
format_stamp: "Formatted at 2026-08-06 23:52:01 on dist-test-slave-nj21"
server_key: "ff75cf1a02e2e7aa63667c7da14e3dc1"
server_key_iv: "3f4712899b94670162b4e08205c6722c"
server_key_version: "encryptionkey@0"
I20260806 23:52:01.209724 16160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 23:52:01.212654 16189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:01.214133 16160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:52:01.214260 16160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "1d9540cd8ae74432871fae5fa25bbdb6"
format_stamp: "Formatted at 2026-08-06 23:52:01 on dist-test-slave-nj21"
server_key: "ff75cf1a02e2e7aa63667c7da14e3dc1"
server_key_iv: "3f4712899b94670162b4e08205c6722c"
server_key_version: "encryptionkey@0"
I20260806 23:52:01.214395 16160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:01.239014 16160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:01.243222 16160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:01.243441 16160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:01.244288 16160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:01.245363 16160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:01.245417 16160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:01.245466 16160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:01.245590 16160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:01.255004 16160 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:41217
I20260806 23:52:01.255043 16373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:41217 every 8 connection(s)
I20260806 23:52:01.256031 16160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:01.264878 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 16160
I20260806 23:52:01.265053 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:01.265398 11142 external_mini_cluster.cc:1428] Setting key d55fe53028c8cd80494c56578b6417eb
Aug 06 23:52:01 dist-test-slave-nj21 krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:01.271941 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:01.258385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:52479 (local address 127.10.225.190:34197)
0806 23:52:01.258592 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:01.258597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:01.259789 (+ 1192us) server_negotiation.cc:407] Connection header received
0806 23:52:01.261282 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:01.261285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:01.261346 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:01.261426 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:01.262716 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.263254 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.264482 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.264709 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.264812 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:01.267417 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:01.267420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:01.267422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:01.267447 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:01.269000 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:01.269794 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:01.269818 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:01.269820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:01.269934 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:01.270549 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:01.270553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:01.270554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:01.270792 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:01.270902 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:01.271636 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 23:52:01.271802 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 23:52:01.273239 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:01.258936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34197 (local address 127.10.225.130:52479)
0806 23:52:01.259601 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:01.259653 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:01.261057 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:01.261624 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:01.261632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:01.262034 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:01.262572 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:01.262585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.263376 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.263378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:01.264332 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:01.264343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.264869 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.264873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:01.265052 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:01.265661 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:01.265678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:01.267257 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:01.269219 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:01.269229 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:01.269250 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:01.269674 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:01.270091 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:01.270096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:01.270097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:01.270392 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:01.270953 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:01.270961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:01.271324 (+ 363us) client_negotiation.cc:769] Sending connection context
0806 23:52:01.271546 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 23:52:01.272013 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":169,"threads_started":1}
I20260806 23:52:01.274583 16374 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34197
I20260806 23:52:01.274890 16374 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:01.275560 16374 heartbeater.cc:511] Master 127.10.225.190:34197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:52:01.277391 15835 ts_manager.cc:194] Registered new tserver with Master: 1d9540cd8ae74432871fae5fa25bbdb6 (127.10.225.130:41217)
I20260806 23:52:01.278257 15835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:52479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:52:01.297168 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34197
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:01.431180 16384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:01.431493 16384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:01.431540 16384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:01.435935 16384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:01.436021 16384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:01.436102 16384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:01.443251 16384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:01.445212 16384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:01.446538 16384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:01.456844 16393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:01.456844 16399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:01.456845 16390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:01.457106 16384 server_base.cc:1034] running on GCE node
I20260806 23:52:01.457517 16384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:01.458201 16384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:01.459395 16384 hybrid_clock.cc:648] HybridClock initialized: now 1786060321459376 us; error 30 us; skew 500 ppm
Aug 06 23:52:01 dist-test-slave-nj21 krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:01.462327 16384 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:52:01.463312 16384 webserver.cc:460] Webserver started at http://127.10.225.131:46647/ using document root <none> and password file <none>
I20260806 23:52:01.464380 16384 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:01.464452 16384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:01.464654 16384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:01.466941 16384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "fa251bce2dd94dc3b7e7630fac40099c"
format_stamp: "Formatted at 2026-08-06 23:52:01 on dist-test-slave-nj21"
server_key: "a6998ac285a9df43e73d77a8b492c184"
server_key_iv: "9419c2c152647e3d7b121ad7226cc655"
server_key_version: "encryptionkey@0"
I20260806 23:52:01.467512 16384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "fa251bce2dd94dc3b7e7630fac40099c"
format_stamp: "Formatted at 2026-08-06 23:52:01 on dist-test-slave-nj21"
server_key: "a6998ac285a9df43e73d77a8b492c184"
server_key_iv: "9419c2c152647e3d7b121ad7226cc655"
server_key_version: "encryptionkey@0"
I20260806 23:52:01.471719 16384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 23:52:01.474531 16409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:01.477016 16384 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:52:01.477248 16384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "fa251bce2dd94dc3b7e7630fac40099c"
format_stamp: "Formatted at 2026-08-06 23:52:01 on dist-test-slave-nj21"
server_key: "a6998ac285a9df43e73d77a8b492c184"
server_key_iv: "9419c2c152647e3d7b121ad7226cc655"
server_key_version: "encryptionkey@0"
I20260806 23:52:01.477419 16384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:01.501030 16384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:01.504688 16384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:01.504922 16384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:01.505749 16384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:01.506733 16384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:01.506789 16384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:01.506842 16384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:01.506873 16384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:01.516357 16384 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:36357
I20260806 23:52:01.516386 16599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:36357 every 8 connection(s)
I20260806 23:52:01.517356 16384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:01.527519 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 16384
I20260806 23:52:01.527663 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:01.527998 11142 external_mini_cluster.cc:1428] Setting key 8cb3a0e8af83f569cd175d829eb8ebae
Aug 06 23:52:01 dist-test-slave-nj21 krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:01.535763 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:01.519600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:50025 (local address 127.10.225.190:34197)
0806 23:52:01.519883 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:01.519888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:01.521294 (+ 1406us) server_negotiation.cc:407] Connection header received
0806 23:52:01.523240 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:01.523244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:01.523417 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:01.523543 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:01.525895 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.526488 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.528581 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.528854 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.528961 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:01.532005 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:01.532008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:01.532010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:01.532038 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:01.533343 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:01.533880 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:01.533883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:01.533885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:01.533971 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:01.534306 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:01.534308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:01.534309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:01.534492 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:01.534561 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:01.535298 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 23:52:01.535526 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260806 23:52:01.536322 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:01.520316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34197 (local address 127.10.225.131:50025)
0806 23:52:01.521069 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:01.521132 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:01.522933 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:01.523869 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:01.523880 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:01.524698 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:01.525572 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:01.525596 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.526672 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.526676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:01.528422 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:01.528444 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.529029 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.529034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:01.529261 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:01.529980 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:01.530002 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:01.531775 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:01.533472 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:01.533478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:01.533493 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:01.533761 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:01.534071 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:01.534074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:01.534075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:01.534229 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:01.534564 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:01.534570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:01.534887 (+ 317us) client_negotiation.cc:769] Sending connection context
0806 23:52:01.535054 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 23:52:01.535351 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":185,"threads_started":1}
I20260806 23:52:01.537801 16600 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34197
I20260806 23:52:01.538070 16600 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:01.538677 16600 heartbeater.cc:511] Master 127.10.225.190:34197 requested a full tablet report, sending...
I20260806 23:52:01.540300 15835 ts_manager.cc:194] Registered new tserver with Master: fa251bce2dd94dc3b7e7630fac40099c (127.10.225.131:36357)
I20260806 23:52:01.541175 15835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:50025
I20260806 23:52:01.543322 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 23:52:01 dist-test-slave-nj21 krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060321, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 23:52:01 dist-test-slave-nj21 krb5kdc[15758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060321, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:01.642827 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:01.631896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.10.225.190:34197)
0806 23:52:01.632056 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:01.632061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:01.632202 (+ 141us) server_negotiation.cc:407] Connection header received
0806 23:52:01.632361 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:01.632364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:01.632421 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:01.632503 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:01.633299 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.634042 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.634933 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:01.635261 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:01.635360 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:01.638487 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:01.638490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:01.638492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:01.638523 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:01.640610 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:01.641047 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:01.641050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:01.641051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:01.641131 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:01.641627 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:01.641633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:01.641634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:01.642007 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:01.642157 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:01.642551 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 23:52:01.642688 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 23:52:01.643280 15833 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42200
I20260806 23:52:01.644695 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 15942
I20260806 23:52:01.679592 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 16160
I20260806 23:52:01.698979 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 16384
I20260806 23:52:01.711813 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 15774
2026-08-06T23:52:01Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2588 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:52:03 dist-test-slave-nj21 krb5kdc[16618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:52:03 dist-test-slave-nj21 krb5kdc[16618](info): set up 2 sockets
Aug 06 23:52:03 dist-test-slave-nj21 krb5kdc[16618](info): commencing operation
krb5kdc: starting...
W20260806 23:52:04.235438 11142 mini_kdc.cc:121] Time spent starting KDC: real 2.488s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:04 dist-test-slave-nj21 krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:52:04.384286 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35533
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:04.508939 16635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:04.509253 16635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:04.509296 16635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:04.513409 16635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:04.513479 16635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:04.513499 16635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:04.513514 16635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:04.513528 16635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:04.519765 16635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:04.521508 16635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:04.522967 16635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:52:04.533113 16635 server_base.cc:1034] running on GCE node
W20260806 23:52:04.533069 16646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:04.533069 16648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:04.533088 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:04.533942 16635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:04.535183 16635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:04.536415 16635 hybrid_clock.cc:648] HybridClock initialized: now 1786060324536353 us; error 66 us; skew 500 ppm
Aug 06 23:52:04 dist-test-slave-nj21 krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:04.540169 16635 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:52:04.541326 16635 webserver.cc:460] Webserver started at http://127.10.225.190:37635/ using document root <none> and password file <none>
I20260806 23:52:04.541965 16635 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:04.542047 16635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:04.542286 16635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:04.544580 16635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "6c3b86fe44f946b2ac14bb852959c8a8"
format_stamp: "Formatted at 2026-08-06 23:52:04 on dist-test-slave-nj21"
server_key: "c46b988b9985f9708cdf0893b30b7c90"
server_key_iv: "107089fea8feab392e39c29c8489a78f"
server_key_version: "encryptionkey@0"
I20260806 23:52:04.545157 16635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "6c3b86fe44f946b2ac14bb852959c8a8"
format_stamp: "Formatted at 2026-08-06 23:52:04 on dist-test-slave-nj21"
server_key: "c46b988b9985f9708cdf0893b30b7c90"
server_key_iv: "107089fea8feab392e39c29c8489a78f"
server_key_version: "encryptionkey@0"
I20260806 23:52:04.549180 16635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:52:04.551756 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:04.553143 16635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:04.553272 16635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "6c3b86fe44f946b2ac14bb852959c8a8"
format_stamp: "Formatted at 2026-08-06 23:52:04 on dist-test-slave-nj21"
server_key: "c46b988b9985f9708cdf0893b30b7c90"
server_key_iv: "107089fea8feab392e39c29c8489a78f"
server_key_version: "encryptionkey@0"
I20260806 23:52:04.553378 16635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:04.574054 16635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:04.578375 16635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:04.578637 16635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:04.588049 16635 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:35533
I20260806 23:52:04.588060 16740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:35533 every 8 connection(s)
I20260806 23:52:04.589212 16635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:04.592334 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:04.593463 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 16635
I20260806 23:52:04.593602 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:04.593914 11142 external_mini_cluster.cc:1428] Setting key ee41b2a1b3afd35aa6f522b9992156ba
I20260806 23:52:04.599936 16742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8: Bootstrap starting.
I20260806 23:52:04.602372 16742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:04.603159 16742 log.cc:824] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:04.605207 16742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8: No bootstrap required, opened a new log
Aug 06 23:52:04 dist-test-slave-nj21 krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:04.609236 16742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3b86fe44f946b2ac14bb852959c8a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35533 } }
I20260806 23:52:04.609575 16742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:04.609637 16742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3b86fe44f946b2ac14bb852959c8a8, State: Initialized, Role: FOLLOWER
I20260806 23:52:04.610168 16742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3b86fe44f946b2ac14bb852959c8a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35533 } }
I20260806 23:52:04.610308 16742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:04.610373 16742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:04.610498 16742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:04.611491 16742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3b86fe44f946b2ac14bb852959c8a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35533 } }
I20260806 23:52:04.611881 16742 leader_election.cc:302] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c3b86fe44f946b2ac14bb852959c8a8; no voters:
I20260806 23:52:04.612229 16742 leader_election.cc:288] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:04.613343 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:04.595396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.10.225.190:35533)
0806 23:52:04.595884 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:04.595900 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:04.595942 (+ 42us) server_negotiation.cc:407] Connection header received
0806 23:52:04.597085 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:04.597133 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:04.597885 (+ 752us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:04.598467 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:04.600486 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:04.601682 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:04.602834 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:04.603623 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:04.603883 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:04.606874 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:04.606883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:04.606904 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:04.606941 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:04.609947 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:04.610459 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:04.610466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:04.610471 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:04.610622 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:04.610987 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:04.610991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:04.610993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:04.611459 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:04.611656 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:04.612001 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 23:52:04.612339 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":155,"threads_started":1}
I20260806 23:52:04.613655 16742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:04.612617 16758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:04.614871 16758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c3b86fe44f946b2ac14bb852959c8a8, State: Running, Role: LEADER
I20260806 23:52:04.615228 16758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3b86fe44f946b2ac14bb852959c8a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35533 } }
I20260806 23:52:04.617201 16754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c3b86fe44f946b2ac14bb852959c8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3b86fe44f946b2ac14bb852959c8a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35533 } } }
I20260806 23:52:04.617244 16766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3b86fe44f946b2ac14bb852959c8a8. Latest consensus state: current_term: 1 leader_uuid: "6c3b86fe44f946b2ac14bb852959c8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3b86fe44f946b2ac14bb852959c8a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35533 } } }
I20260806 23:52:04.617340 16754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:04.617357 16766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:04.617844 16783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:04.620923 16783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:04.628590 16783 catalog_manager.cc:1347] Generated new cluster ID: 6133a19f150c40fa89320a89922cbfa9
I20260806 23:52:04.628687 16783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:04.649680 16783 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:04.650748 16783 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:04.664083 16783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c3b86fe44f946b2ac14bb852959c8a8: Generated new TSK 0
I20260806 23:52:04.664875 16783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:52:04.704869 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35533
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:04.848337 16792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:04.848644 16792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:04.848690 16792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:04.853070 16792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:04.853144 16792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:04.853214 16792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:04.858781 16792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:04.860021 16792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:04.861127 16792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:04.871271 16804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:04.871416 16792 server_base.cc:1034] running on GCE node
W20260806 23:52:04.872493 16800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:04.873095 16806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:04.873602 16792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:04.874336 16792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:04.875519 16792 hybrid_clock.cc:648] HybridClock initialized: now 1786060324875489 us; error 40 us; skew 500 ppm
Aug 06 23:52:04 dist-test-slave-nj21 krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:04.878779 16792 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:52:04.879827 16792 webserver.cc:460] Webserver started at http://127.10.225.129:32809/ using document root <none> and password file <none>
I20260806 23:52:04.880407 16792 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:04.880476 16792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:04.880676 16792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:04.882807 16792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "04830a87028f4360b81801f65f0742b9"
format_stamp: "Formatted at 2026-08-06 23:52:04 on dist-test-slave-nj21"
server_key: "e2706f608470a568681b5ec5a5b86c5b"
server_key_iv: "17c2f963401bcb8932c78cd13ac53206"
server_key_version: "encryptionkey@0"
I20260806 23:52:04.883373 16792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "04830a87028f4360b81801f65f0742b9"
format_stamp: "Formatted at 2026-08-06 23:52:04 on dist-test-slave-nj21"
server_key: "e2706f608470a568681b5ec5a5b86c5b"
server_key_iv: "17c2f963401bcb8932c78cd13ac53206"
server_key_version: "encryptionkey@0"
I20260806 23:52:04.887233 16792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:04.889943 16817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:04.891222 16792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:04.891335 16792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "04830a87028f4360b81801f65f0742b9"
format_stamp: "Formatted at 2026-08-06 23:52:04 on dist-test-slave-nj21"
server_key: "e2706f608470a568681b5ec5a5b86c5b"
server_key_iv: "17c2f963401bcb8932c78cd13ac53206"
server_key_version: "encryptionkey@0"
I20260806 23:52:04.891466 16792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:04.908843 16792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:04.912886 16792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:04.913087 16792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:04.913785 16792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:04.914774 16792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:04.914832 16792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:04.914876 16792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:04.914893 16792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:04.925019 16792 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:34629
I20260806 23:52:04.925063 16998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:34629 every 8 connection(s)
I20260806 23:52:04.926018 16792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:04.935189 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 16792
I20260806 23:52:04.935333 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:04.935648 11142 external_mini_cluster.cc:1428] Setting key c85a454aae5a8f42423174ef8f924671
Aug 06 23:52:04 dist-test-slave-nj21 krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060324, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:04.941684 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:04.928014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:43199 (local address 127.10.225.190:35533)
0806 23:52:04.928147 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:04.928151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:04.929017 (+ 866us) server_negotiation.cc:407] Connection header received
0806 23:52:04.930458 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:04.930461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:04.930523 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:04.930601 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:04.931860 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:04.932433 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:04.933562 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:04.933783 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:04.933871 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:04.936729 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:04.936734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:04.936737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:04.936782 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:04.938578 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:04.939212 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:04.939216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:04.939217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:04.939453 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:04.940030 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:04.940034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:04.940036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:04.940313 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:04.940437 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:04.941353 (+ 916us) server_negotiation.cc:299] Negotiation successful
0806 23:52:04.941510 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:52:04.943123 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:04.928353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35533 (local address 127.10.225.129:43199)
0806 23:52:04.928858 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:04.928900 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:04.930242 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:04.930787 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:04.930794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:04.931162 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:04.931721 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:04.931738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:04.932561 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:04.932564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:04.933441 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:04.933450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:04.933931 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:04.933936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:04.934099 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:04.934671 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:04.934691 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:04.936427 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:04.938709 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:04.938716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:04.938735 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:04.939096 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:04.939638 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:04.939642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:04.939644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:04.939874 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:04.940553 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:04.940564 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:04.941002 (+ 438us) client_negotiation.cc:769] Sending connection context
0806 23:52:04.941251 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 23:52:04.941703 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":120,"threads_started":1}
I20260806 23:52:04.944833 16999 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35533
I20260806 23:52:04.945303 16999 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:04.946197 16999 heartbeater.cc:511] Master 127.10.225.190:35533 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:52:04.948222 16692 ts_manager.cc:194] Registered new tserver with Master: 04830a87028f4360b81801f65f0742b9 (127.10.225.129:34629)
I20260806 23:52:04.949432 16692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:43199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:52:04.970723 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35533
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:05.106523 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:05.106829 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:05.106895 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:05.111070 17006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:05.111145 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:05.111227 17006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:05.117055 17006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:05.118436 17006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:05.119604 17006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:05.131942 17019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:05.131939 17017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:05.132172 17006 server_base.cc:1034] running on GCE node
W20260806 23:52:05.132063 17016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:05.132833 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:05.133773 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:05.134959 17006 hybrid_clock.cc:648] HybridClock initialized: now 1786060325134927 us; error 43 us; skew 500 ppm
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:05.139572 17006 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:52:05.141237 17006 webserver.cc:460] Webserver started at http://127.10.225.130:33191/ using document root <none> and password file <none>
I20260806 23:52:05.142030 17006 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:05.142112 17006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:05.142381 17006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:05.145247 17006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "cc5cb5ace6104a8a828b6f9574251d45"
format_stamp: "Formatted at 2026-08-06 23:52:05 on dist-test-slave-nj21"
server_key: "d0fc6f6fd43ac8b3d4d665b47b6b42b6"
server_key_iv: "52dd71a54d62f25eca9120d133cccf5c"
server_key_version: "encryptionkey@0"
I20260806 23:52:05.145936 17006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "cc5cb5ace6104a8a828b6f9574251d45"
format_stamp: "Formatted at 2026-08-06 23:52:05 on dist-test-slave-nj21"
server_key: "d0fc6f6fd43ac8b3d4d665b47b6b42b6"
server_key_iv: "52dd71a54d62f25eca9120d133cccf5c"
server_key_version: "encryptionkey@0"
I20260806 23:52:05.150945 17006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:05.153824 17030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:05.155361 17006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:05.155486 17006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "cc5cb5ace6104a8a828b6f9574251d45"
format_stamp: "Formatted at 2026-08-06 23:52:05 on dist-test-slave-nj21"
server_key: "d0fc6f6fd43ac8b3d4d665b47b6b42b6"
server_key_iv: "52dd71a54d62f25eca9120d133cccf5c"
server_key_version: "encryptionkey@0"
I20260806 23:52:05.155617 17006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:05.168780 17006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:05.172448 17006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:05.172653 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:05.173341 17006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:05.174271 17006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:05.174332 17006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:05.174392 17006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:05.174417 17006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:05.184993 17006 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:43399
I20260806 23:52:05.185011 17213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:43399 every 8 connection(s)
I20260806 23:52:05.186002 17006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:05.189491 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 17006
I20260806 23:52:05.189669 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:05.189992 11142 external_mini_cluster.cc:1428] Setting key fad64545fe10e299fefc4f9e5141689c
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:52:05.203192 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:05.187977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:58591 (local address 127.10.225.190:35533)
0806 23:52:05.188116 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:05.188119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:05.189283 (+ 1164us) server_negotiation.cc:407] Connection header received
0806 23:52:05.191232 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:05.191236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:05.191294 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:05.191366 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:05.193118 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.193834 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.195091 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.195328 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.195420 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:05.198598 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:05.198602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:05.198605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:05.198637 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:05.200330 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:05.200998 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:05.201002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:05.201004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:05.201097 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:05.201495 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:05.201498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:05.201500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:05.201756 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:05.201860 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:05.202796 (+ 936us) server_negotiation.cc:299] Negotiation successful
0806 23:52:05.203004 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:52:05.204310 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:05.188524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35533 (local address 127.10.225.130:58591)
0806 23:52:05.189103 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:05.189159 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:05.191006 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:05.191621 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:05.191631 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:05.192204 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:05.192938 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:05.192958 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.193960 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.193964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:05.194964 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:05.194974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.195478 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.195482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:05.195699 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:05.196385 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:05.196407 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:05.198362 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:05.200474 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:05.200482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:05.200502 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:05.200886 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:05.201199 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:05.201202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:05.201204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:05.201397 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:05.201870 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:05.201877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:05.202234 (+ 357us) client_negotiation.cc:769] Sending connection context
0806 23:52:05.202466 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 23:52:05.202875 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":159,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
I20260806 23:52:05.205958 17214 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35533
I20260806 23:52:05.206266 17214 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:05.206897 17214 heartbeater.cc:511] Master 127.10.225.190:35533 requested a full tablet report, sending...
I20260806 23:52:05.208139 16692 ts_manager.cc:194] Registered new tserver with Master: cc5cb5ace6104a8a828b6f9574251d45 (127.10.225.130:43399)
I20260806 23:52:05.208762 16692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:58591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:52:05.226212 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35533
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:05.351251 17220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:05.351572 17220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:05.351639 17220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:05.355984 17220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:05.356055 17220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:05.356141 17220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:05.361936 17220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:05.363152 17220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:05.364305 17220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:05.374922 17228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:05.374917 17231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:05.374917 17238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:05.375361 17220 server_base.cc:1034] running on GCE node
I20260806 23:52:05.375839 17220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:05.376614 17220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:05.377808 17220 hybrid_clock.cc:648] HybridClock initialized: now 1786060325377767 us; error 52 us; skew 500 ppm
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:05.380983 17220 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:52:05.382401 17220 webserver.cc:460] Webserver started at http://127.10.225.131:38081/ using document root <none> and password file <none>
I20260806 23:52:05.383286 17220 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:05.383378 17220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:05.383704 17220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:05.386840 17220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "3eabe578023e489c81bc9912ff7a60c6"
format_stamp: "Formatted at 2026-08-06 23:52:05 on dist-test-slave-nj21"
server_key: "cd9a41f2c808265db6131bd95ab6c85f"
server_key_iv: "f340a2abd2ab3697adf406adb4611211"
server_key_version: "encryptionkey@0"
I20260806 23:52:05.387840 17220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "3eabe578023e489c81bc9912ff7a60c6"
format_stamp: "Formatted at 2026-08-06 23:52:05 on dist-test-slave-nj21"
server_key: "cd9a41f2c808265db6131bd95ab6c85f"
server_key_iv: "f340a2abd2ab3697adf406adb4611211"
server_key_version: "encryptionkey@0"
I20260806 23:52:05.392613 17220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:52:05.395568 17246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:05.396926 17220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:52:05.397053 17220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "3eabe578023e489c81bc9912ff7a60c6"
format_stamp: "Formatted at 2026-08-06 23:52:05 on dist-test-slave-nj21"
server_key: "cd9a41f2c808265db6131bd95ab6c85f"
server_key_iv: "f340a2abd2ab3697adf406adb4611211"
server_key_version: "encryptionkey@0"
I20260806 23:52:05.397171 17220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:05.410228 17220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:05.413553 17220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:05.413744 17220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:05.414405 17220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:05.415403 17220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:05.415453 17220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:05.415509 17220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:05.415539 17220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:05.425527 17220 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:43915
I20260806 23:52:05.425601 17430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:43915 every 8 connection(s)
I20260806 23:52:05.426476 17220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:05.435432 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 17220
I20260806 23:52:05.435623 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:05.435948 11142 external_mini_cluster.cc:1428] Setting key e7b06bd8e2220c779c3931f3709ce275
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060325, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:05.441677 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:05.428321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:37385 (local address 127.10.225.190:35533)
0806 23:52:05.428497 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:05.428500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:05.429449 (+ 949us) server_negotiation.cc:407] Connection header received
0806 23:52:05.430839 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:05.430841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:05.430893 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:05.430980 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:05.432793 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.433434 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.434891 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.435185 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.435316 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:05.438306 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:05.438310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:05.438312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:05.438338 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:05.439627 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:05.440157 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:05.440160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:05.440162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:05.440240 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:05.440567 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:05.440571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:05.440572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:05.440761 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:05.440856 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:05.441371 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 23:52:05.441505 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 23:52:05.442556 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:05.428697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35533 (local address 127.10.225.131:37385)
0806 23:52:05.429221 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:05.429332 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:05.430615 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:05.431419 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:05.431432 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:05.432020 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:05.432537 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:05.432554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.433651 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.433654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:05.434734 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:05.434746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.435440 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.435444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:05.435834 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:05.436488 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:05.436521 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:05.438152 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:05.439776 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:05.439783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:05.439800 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:05.440060 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:05.440332 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:05.440335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:05.440337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:05.440478 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:05.440864 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:05.440871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:05.441134 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 23:52:05.441314 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 23:52:05.441673 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":122,"threads_started":1}
I20260806 23:52:05.443755 17431 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35533
I20260806 23:52:05.444015 17431 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:05.444584 17431 heartbeater.cc:511] Master 127.10.225.190:35533 requested a full tablet report, sending...
I20260806 23:52:05.445751 16692 ts_manager.cc:194] Registered new tserver with Master: 3eabe578023e489c81bc9912ff7a60c6 (127.10.225.131:43915)
I20260806 23:52:05.446368 16692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:37385
I20260806 23:52:05.450155 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:52:05.480624 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:05.469682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.10.225.129:34629)
0806 23:52:05.470062 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:05.470069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:05.470094 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:52:05.470149 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:05.470155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:05.470356 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:05.470488 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:05.471153 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.472023 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.472938 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.473338 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.473435 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:05.476165 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:05.476177 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:05.476200 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:05.476234 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:05.477845 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:05.478437 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:05.478449 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:05.478450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:05.478647 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:05.479033 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:05.479036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:05.479038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:05.479573 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:05.479893 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:05.480254 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 23:52:05.480427 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":148,"threads_started":1}
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:05.493808 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:05.484436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.10.225.190:35533)
0806 23:52:05.484549 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:05.484552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:05.484566 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:05.484687 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:05.484689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:05.484731 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:05.484803 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:05.485531 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.486078 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.487184 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:05.487501 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:05.487601 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:05.490325 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:05.490329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:05.490332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:05.490363 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:05.492055 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:05.492564 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:05.492568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:05.492569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:05.492671 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:05.492916 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:05.492918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:05.492919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:05.493180 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:05.493294 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:05.493529 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 23:52:05.493640 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 23:52:05.494297 16692 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56636
I20260806 23:52:05.495059 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 16792
I20260806 23:52:05.510219 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 17006
I20260806 23:52:05.523069 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 17220
I20260806 23:52:05.536698 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 16635
2026-08-06T23:52:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3823 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[17449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[17449](info): set up 2 sockets
Aug 06 23:52:05 dist-test-slave-nj21 krb5kdc[17449](info): commencing operation
krb5kdc: starting...
W20260806 23:52:06.704560 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:06 dist-test-slave-nj21 krb5kdc[17449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:52:06.847040 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:39423
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:39423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:06.974022 17475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:06.974413 17475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:06.974464 17475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:06.979225 17475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:06.979303 17475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:06.979323 17475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:06.979339 17475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:06.979353 17475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:06.985123 17475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:39423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:39423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:06.986258 17475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:06.987285 17475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:52:06.996363 17475 server_base.cc:1034] running on GCE node
W20260806 23:52:06.996352 17484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:06.996352 17485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:06.996587 17487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:06.997198 17475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:06.998389 17475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:06.999568 17475 hybrid_clock.cc:648] HybridClock initialized: now 1786060326999542 us; error 39 us; skew 500 ppm
Aug 06 23:52:07 dist-test-slave-nj21 krb5kdc[17449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:07.009775 17475 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:52:07.010671 17475 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:07.010744 17475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:07.010968 17475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:07.013223 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "de6e1075e02d018ff447fce11add94a2"
server_key_iv: "629b6be2e1f0be632fab6219f9161f66"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.013784 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "de6e1075e02d018ff447fce11add94a2"
server_key_iv: "629b6be2e1f0be632fab6219f9161f66"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.017757 17475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:52:07.021360 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.023342 17475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:52:07.023564 17475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "de6e1075e02d018ff447fce11add94a2"
server_key_iv: "629b6be2e1f0be632fab6219f9161f66"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.023766 17475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:07.046314 17475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:07.049734 17475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:07.049950 17475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:07.057693 17475 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:39423
I20260806 23:52:07.057989 17575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:39423 every 8 connection(s)
I20260806 23:52:07.058416 17475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:07.062425 17577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:07.066238 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 17475
I20260806 23:52:07.066421 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:07.066782 11142 external_mini_cluster.cc:1428] Setting key f4443a5fca072ba5de6dd6cb30f7be88
I20260806 23:52:07.071089 17577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27: Bootstrap starting.
I20260806 23:52:07.074574 17577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:07.075485 17577 log.cc:824] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:52:07 dist-test-slave-nj21 krb5kdc[17449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:07.077850 17577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27: No bootstrap required, opened a new log
I20260806 23:52:07.081827 17577 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39423 } }
I20260806 23:52:07.082090 17577 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:07.082159 17577 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9af0b6ee7794fb7a8c6400ccdef6b27, State: Initialized, Role: FOLLOWER
I20260806 23:52:07.082656 17577 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39423 } }
I20260806 23:52:07.082808 17577 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:07.082877 17577 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:07.082974 17577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:07.083909 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.068918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.10.225.190:39423)
0806 23:52:07.069474 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:07.069492 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:07.069539 (+ 47us) server_negotiation.cc:407] Connection header received
0806 23:52:07.070431 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:07.070454 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:07.070866 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:07.071206 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:07.072323 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.073582 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.074590 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.075145 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.075308 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.077943 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:07.077949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:07.077970 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:07.078005 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:07.080079 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.080678 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.080685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.080692 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.080844 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.081149 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.081152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.081154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.081729 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:07.081975 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:07.082308 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 23:52:07.082752 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":159,"threads_started":1}
I20260806 23:52:07.084107 17577 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39423 } }
I20260806 23:52:07.084498 17577 leader_election.cc:302] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9af0b6ee7794fb7a8c6400ccdef6b27; no voters:
I20260806 23:52:07.084863 17577 leader_election.cc:288] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:07.085126 17584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:07.085451 17584 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [term 1 LEADER]: Becoming Leader. State: Replica: d9af0b6ee7794fb7a8c6400ccdef6b27, State: Running, Role: LEADER
I20260806 23:52:07.085795 17584 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39423 } }
I20260806 23:52:07.086337 17577 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:07.087664 17585 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39423 } } }
I20260806 23:52:07.087857 17585 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:07.088207 17586 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9af0b6ee7794fb7a8c6400ccdef6b27. Latest consensus state: current_term: 1 leader_uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9af0b6ee7794fb7a8c6400ccdef6b27" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39423 } } }
I20260806 23:52:07.088316 17586 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27 [sys.catalog]: This master's current role is: LEADER
W20260806 23:52:07.091828 17607 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:52:07.091907 17607 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:52:07.091995 17599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:07.092690 17599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:07.099928 17599 catalog_manager.cc:1347] Generated new cluster ID: be0439e4da314e259d3b75fabfa00945
I20260806 23:52:07.100060 17599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:07.111930 17599 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:07.113050 17599 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:07.119148 17599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27: Generated new TSK 0
I20260806 23:52:07.119987 17599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:52:07.157341 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:39423
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:07.290370 17616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:07.290699 17616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:07.290771 17616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:07.295558 17616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:07.295645 17616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:07.295761 17616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:07.301576 17616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:39423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:07.302800 17616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:07.303933 17616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:07.313882 17627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:07.313957 17625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:07.313957 17630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:07.314297 17616 server_base.cc:1034] running on GCE node
I20260806 23:52:07.314788 17616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:07.315488 17616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:07.316679 17616 hybrid_clock.cc:648] HybridClock initialized: now 1786060327316654 us; error 42 us; skew 500 ppm
Aug 06 23:52:07 dist-test-slave-nj21 krb5kdc[17449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:07.319892 17616 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:52:07.321003 17616 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:07.321089 17616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:07.321347 17616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:07.324036 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "9a445f91c5a343778021e10686572c61"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "030e1590d960c46f9d2ffaccf16ad5ea"
server_key_iv: "8fe3dea786bc608c96bc41f5470cb334"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.324688 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "9a445f91c5a343778021e10686572c61"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "030e1590d960c46f9d2ffaccf16ad5ea"
server_key_iv: "8fe3dea786bc608c96bc41f5470cb334"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.329783 17616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 23:52:07.333187 17641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.335047 17616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:52:07.335224 17616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "9a445f91c5a343778021e10686572c61"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "030e1590d960c46f9d2ffaccf16ad5ea"
server_key_iv: "8fe3dea786bc608c96bc41f5470cb334"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.335371 17616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:07.357779 17616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:07.361362 17616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:07.361555 17616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:07.362187 17616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:07.363067 17616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:07.363127 17616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.363183 17616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:07.363211 17616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.373804 17616 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:37803
I20260806 23:52:07.373862 17829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:37803 every 8 connection(s)
I20260806 23:52:07.374840 17616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:07.377677 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 17616
I20260806 23:52:07.377816 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:07.378108 11142 external_mini_cluster.cc:1428] Setting key 29243fbaf34aee45b705d0e6db40ffc0
Aug 06 23:52:07 dist-test-slave-nj21 krb5kdc[17449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:52:07.391970 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.377521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:58237 (local address 127.10.225.190:39423)
0806 23:52:07.377778 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:07.377784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:07.379239 (+ 1455us) server_negotiation.cc:407] Connection header received
0806 23:52:07.380204 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:07.380207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:07.380262 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:07.380335 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:07.381536 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.382087 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.383206 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.383437 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.383524 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.386874 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:07.386878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:07.386881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:07.386914 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:07.388791 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.389574 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.389577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.389578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.389667 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.390204 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.390207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.390208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.390472 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:07.390569 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:07.391479 (+ 910us) server_negotiation.cc:299] Negotiation successful
0806 23:52:07.391778 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260806 23:52:07.393118 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.378089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:39423 (local address 127.10.225.129:58237)
0806 23:52:07.379074 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:07.379130 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:07.380009 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:07.380509 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:07.380516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:07.380881 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:07.381396 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.381410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.382207 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.382210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.383090 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.383100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.383583 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.383587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.383780 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.384550 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:07.384576 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:07.386657 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:07.388979 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:07.388988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:07.389012 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:07.389424 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:07.389820 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:07.389824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:07.389827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:07.390048 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:07.390655 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:07.390663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:07.391115 (+ 452us) client_negotiation.cc:769] Sending connection context
0806 23:52:07.391373 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 23:52:07.391992 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":145,"threads_started":1}
I20260806 23:52:07.394616 17830 heartbeater.cc:347] Connected to a master server at 127.10.225.190:39423
I20260806 23:52:07.394922 17830 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:07.395526 17830 heartbeater.cc:511] Master 127.10.225.190:39423 requested a full tablet report, sending...
I20260806 23:52:07.397907 17521 ts_manager.cc:194] Registered new tserver with Master: 9a445f91c5a343778021e10686572c61 (127.10.225.129:37803)
I20260806 23:52:07.399542 17521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:58237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:52:07.409238 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:39423
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:07.538823 17841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:07.539114 17841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:07.539163 17841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:07.543840 17841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:07.543916 17841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:07.543984 17841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:07.551023 17841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:39423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:07.552635 17841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:07.553794 17841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:07.564316 17852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:07.564316 17854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:07.564317 17851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:07.564368 17841 server_base.cc:1034] running on GCE node
I20260806 23:52:07.564952 17841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:07.565644 17841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:07.566847 17841 hybrid_clock.cc:648] HybridClock initialized: now 1786060327566824 us; error 33 us; skew 500 ppm
Aug 06 23:52:07 dist-test-slave-nj21 krb5kdc[17449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:07.570477 17841 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:52:07.571511 17841 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:07.571699 17841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:07.572047 17841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:07.574290 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "bfc2b9d12165499f93eec5cc1a432f4a"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "0dd963dba46e99dbb0ba4a9dbd302246"
server_key_iv: "4da4d0cebbf4b15cbe9c21a6090266fc"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.574836 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "bfc2b9d12165499f93eec5cc1a432f4a"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "0dd963dba46e99dbb0ba4a9dbd302246"
server_key_iv: "4da4d0cebbf4b15cbe9c21a6090266fc"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.579280 17841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260806 23:52:07.582412 17860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.583900 17841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:07.584007 17841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "bfc2b9d12165499f93eec5cc1a432f4a"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "0dd963dba46e99dbb0ba4a9dbd302246"
server_key_iv: "4da4d0cebbf4b15cbe9c21a6090266fc"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.584105 17841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:07.598789 17841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:07.602159 17841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:07.602452 17841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:07.603060 17841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:07.603951 17841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:07.604001 17841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.604040 17841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:07.604061 17841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.615357 18034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:43137 every 8 connection(s)
I20260806 23:52:07.615370 17841 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:43137
I20260806 23:52:07.616272 17841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:07.618355 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 17841
I20260806 23:52:07.618463 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:07.618731 11142 external_mini_cluster.cc:1428] Setting key 27f349f18e44b3f19a9060b7971a086c
Aug 06 23:52:07 dist-test-slave-nj21 krb5kdc[17449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:52:07.631603 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.618143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:44417 (local address 127.10.225.190:39423)
0806 23:52:07.618300 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:07.618304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:07.619250 (+ 946us) server_negotiation.cc:407] Connection header received
0806 23:52:07.620773 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:07.620776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:07.620835 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:07.620911 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:07.622164 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.622691 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.624066 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.624297 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.624384 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.627082 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:07.627086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:07.627088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:07.627113 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:07.628800 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.629579 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.629583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.629584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.629673 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.630264 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.630268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.630269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.630524 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:07.630622 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:07.631297 (+ 675us) server_negotiation.cc:299] Negotiation successful
0806 23:52:07.631442 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
I20260806 23:52:07.632615 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.618577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:39423 (local address 127.10.225.130:44417)
0806 23:52:07.619084 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:07.619151 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:07.620537 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:07.621095 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:07.621101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:07.621469 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:07.621999 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.622020 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.622819 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.622823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.623950 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.623960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.624438 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.624442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.624602 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.625187 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:07.625204 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:07.626906 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:07.629025 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:07.629034 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:07.629060 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:07.629432 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:07.629770 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:07.629774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:07.629776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:07.630094 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:07.630675 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:07.630682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:07.630999 (+ 317us) client_negotiation.cc:769] Sending connection context
0806 23:52:07.631229 (+ 230us) client_negotiation.cc:240] Negotiation successful
0806 23:52:07.631557 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260806 23:52:07.633852 18035 heartbeater.cc:347] Connected to a master server at 127.10.225.190:39423
I20260806 23:52:07.634126 18035 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:07.634699 18035 heartbeater.cc:511] Master 127.10.225.190:39423 requested a full tablet report, sending...
I20260806 23:52:07.636076 17521 ts_manager.cc:194] Registered new tserver with Master: bfc2b9d12165499f93eec5cc1a432f4a (127.10.225.130:43137)
I20260806 23:52:07.636780 17521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:44417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:52:07.648967 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:39423
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:07.791158 18041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:07.791497 18041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:07.791548 18041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:07.796047 18041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:07.796124 18041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:07.796192 18041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:07.802878 18041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:39423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:07.804193 18041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:07.805286 18041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:07.815106 18048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:07.815140 18053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:07.815109 18050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:07.815569 18041 server_base.cc:1034] running on GCE node
I20260806 23:52:07.816036 18041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:07.816720 18041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:07.817960 18041 hybrid_clock.cc:648] HybridClock initialized: now 1786060327817936 us; error 48 us; skew 500 ppm
Aug 06 23:52:07 dist-test-slave-nj21 krb5kdc[17449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:07.820994 18041 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:52:07.821805 18041 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:07.821872 18041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:07.822074 18041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:07.824311 18041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "34a402917de0493781fa5b49bd3cdd57"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "4c12e0790eeb6da8aecc28d2bc5d78b4"
server_key_iv: "117ad69e3579e3ea51a75dc115bda56b"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.824858 18041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "34a402917de0493781fa5b49bd3cdd57"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "4c12e0790eeb6da8aecc28d2bc5d78b4"
server_key_iv: "117ad69e3579e3ea51a75dc115bda56b"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.830515 18041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 23:52:07.833803 18062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.835330 18041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:07.835450 18041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "34a402917de0493781fa5b49bd3cdd57"
format_stamp: "Formatted at 2026-08-06 23:52:07 on dist-test-slave-nj21"
server_key: "4c12e0790eeb6da8aecc28d2bc5d78b4"
server_key_iv: "117ad69e3579e3ea51a75dc115bda56b"
server_key_version: "encryptionkey@0"
I20260806 23:52:07.835587 18041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:07.846153 18041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:07.850512 18041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:07.850719 18041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:07.851356 18041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:07.852298 18041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:07.852350 18041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.852408 18041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:07.852438 18041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:07.861690 18041 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:34713
I20260806 23:52:07.861739 18238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:34713 every 8 connection(s)
I20260806 23:52:07.862326 18041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:07.868983 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 18041
I20260806 23:52:07.869119 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:07.869669 11142 external_mini_cluster.cc:1428] Setting key 6638ca5324c1478284e602f89677529e
Aug 06 23:52:07 dist-test-slave-nj21 krb5kdc[17449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060327, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:07.877611 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.864331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:59187 (local address 127.10.225.190:39423)
0806 23:52:07.864491 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:07.864495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:07.865368 (+ 873us) server_negotiation.cc:407] Connection header received
0806 23:52:07.866646 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:07.866649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:07.866709 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:07.866783 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:07.868043 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.868579 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.869786 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.870038 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.870127 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.873378 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:07.873382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:07.873385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:07.873421 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:07.874999 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.875927 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.875930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.875932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.876025 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.876417 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.876419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.876421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.876614 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:07.876690 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:07.877346 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 23:52:07.877477 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 23:52:07.878396 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.864696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:39423 (local address 127.10.225.131:59187)
0806 23:52:07.865191 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:07.865245 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:07.866446 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:07.866962 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:07.866969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:07.867357 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:07.867907 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.867924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.868719 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.868721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.869678 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.869691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.870185 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.870189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.870354 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.871043 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:07.871065 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:07.873131 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:07.875160 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:07.875168 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:07.875185 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:07.875777 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:07.876152 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:07.876155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:07.876156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:07.876326 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:07.876705 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:07.876711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:07.876998 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 23:52:07.877176 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 23:52:07.877488 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":116,"threads_started":1}
I20260806 23:52:07.879596 18240 heartbeater.cc:347] Connected to a master server at 127.10.225.190:39423
I20260806 23:52:07.879859 18240 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:07.880363 18240 heartbeater.cc:511] Master 127.10.225.190:39423 requested a full tablet report, sending...
I20260806 23:52:07.881386 17521 ts_manager.cc:194] Registered new tserver with Master: 34a402917de0493781fa5b49bd3cdd57 (127.10.225.131:34713)
I20260806 23:52:07.881955 17521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:59187
I20260806 23:52:07.884748 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:52:07.895627 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.886866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.10.225.190:39423)
0806 23:52:07.887025 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:07.887029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:07.887169 (+ 140us) server_negotiation.cc:407] Connection header received
0806 23:52:07.887351 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:07.887355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:07.887457 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:07.887590 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:07.888573 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.889137 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.890007 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.890238 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.890328 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.892327 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:07.892330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:07.892332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:07.892365 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:07.893704 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.894194 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.894196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.894198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.894274 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:07.894659 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:07.894661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:07.894662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:07.894895 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:07.894998 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:07.895364 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 23:52:07.895489 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 23:52:07.898722 17521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:52:07.902020 17521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:07.919237 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.912440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:43137 (local address 127.0.0.1:39276)
0806 23:52:07.913980 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:07.913995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:07.914086 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:07.914741 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:07.914744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:07.914780 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:07.915036 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.915044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.916565 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.916569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.917500 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.917511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.918611 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.918615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.918750 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.918799 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 23:52:07.918925 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 23:52:07.919060 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1477,"mutex_wait_us":116,"spinlock_wait_cycles":8064,"thread_start_us":188,"threads_started":1}
I20260806 23:52:07.919255 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.912620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.10.225.130:43137)
0806 23:52:07.913340 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:07.913357 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:07.914079 (+ 722us) server_negotiation.cc:407] Connection header received
0806 23:52:07.914244 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:07.914250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:07.914397 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:07.914503 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:07.915161 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.916416 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.917636 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.918488 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.918612 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.918766 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:07.918970 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 23:52:07.919082 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":120,"threads_started":1}
I20260806 23:52:07.919862 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.913416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.10.225.129:37803)
0806 23:52:07.913850 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:07.913857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:07.914137 (+ 280us) server_negotiation.cc:407] Connection header received
0806 23:52:07.914315 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:07.914355 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:07.914705 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:07.914842 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:07.915225 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.916024 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.918213 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.918883 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.918992 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.919109 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:07.919585 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 23:52:07.919717 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":119,"threads_started":1}
I20260806 23:52:07.920158 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.912406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:34713 (local address 127.0.0.1:41572)
0806 23:52:07.913646 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:07.913703 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:07.913845 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:07.914735 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:07.914743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:07.914782 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:07.915079 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.915090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.916566 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.916569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.918672 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.918682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.919795 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.919799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.919885 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.919911 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 23:52:07.919961 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 23:52:07.920024 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":1021,"spinlock_wait_cycles":8832,"thread_start_us":104,"threads_started":1}
I20260806 23:52:07.920209 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.912725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:37803 (local address 127.0.0.1:47688)
0806 23:52:07.914062 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:07.914073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:07.914138 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:07.914983 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:07.914985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:07.915011 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:07.915115 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.915120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.916154 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.916158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.918100 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:07.918106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.919171 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.919174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:07.919271 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.919295 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:52:07.919995 (+ 700us) client_negotiation.cc:240] Negotiation successful
0806 23:52:07.920088 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"mutex_wait_us":916,"thread_start_us":230,"threads_started":1}
I20260806 23:52:07.920794 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:07.913435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.10.225.131:34713)
0806 23:52:07.913921 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:07.913934 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:07.913957 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:52:07.914036 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:07.914041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:07.914372 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:07.914743 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:07.915220 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.916418 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.918841 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:07.919651 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:07.920348 (+ 697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:07.920499 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:07.920547 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 23:52:07.920641 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":152,"threads_started":1}
I20260806 23:52:07.923125 17722 tablet_service.cc:1467] Processing CreateTablet for tablet ccfd4918140c47e3a4bdc07ddbd22456 (DEFAULT_TABLE table=test-table [id=8502440e721f43b6b17e4732c5186e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:07.923118 18141 tablet_service.cc:1467] Processing CreateTablet for tablet ccfd4918140c47e3a4bdc07ddbd22456 (DEFAULT_TABLE table=test-table [id=8502440e721f43b6b17e4732c5186e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:07.923127 17935 tablet_service.cc:1467] Processing CreateTablet for tablet ccfd4918140c47e3a4bdc07ddbd22456 (DEFAULT_TABLE table=test-table [id=8502440e721f43b6b17e4732c5186e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:07.924381 17722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccfd4918140c47e3a4bdc07ddbd22456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:07.924381 17935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccfd4918140c47e3a4bdc07ddbd22456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:07.924396 18141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccfd4918140c47e3a4bdc07ddbd22456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:07.931344 18263 tablet_bootstrap.cc:492] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: Bootstrap starting.
I20260806 23:52:07.932219 18269 tablet_bootstrap.cc:492] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a: Bootstrap starting.
I20260806 23:52:07.932323 18265 tablet_bootstrap.cc:492] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61: Bootstrap starting.
I20260806 23:52:07.934440 18263 tablet_bootstrap.cc:654] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:07.934552 18269 tablet_bootstrap.cc:654] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:07.934702 18265 tablet_bootstrap.cc:654] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:07.935725 18263 log.cc:824] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:07.935750 18269 log.cc:824] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:07.935755 18265 log.cc:824] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:07.937700 18269 tablet_bootstrap.cc:492] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a: No bootstrap required, opened a new log
I20260806 23:52:07.937786 18263 tablet_bootstrap.cc:492] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: No bootstrap required, opened a new log
I20260806 23:52:07.937786 18265 tablet_bootstrap.cc:492] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61: No bootstrap required, opened a new log
I20260806 23:52:07.937918 18269 ts_tablet_manager.cc:1397] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:52:07.937983 18263 ts_tablet_manager.cc:1397] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 23:52:07.937983 18265 ts_tablet_manager.cc:1397] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:52:07.941455 18269 raft_consensus.cc:348] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } }
I20260806 23:52:07.941896 18269 raft_consensus.cc:374] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:07.942015 18269 raft_consensus.cc:729] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc2b9d12165499f93eec5cc1a432f4a, State: Initialized, Role: FOLLOWER
I20260806 23:52:07.941913 18263 raft_consensus.cc:348] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } }
I20260806 23:52:07.942277 18263 raft_consensus.cc:374] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:07.942373 18263 raft_consensus.cc:729] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a402917de0493781fa5b49bd3cdd57, State: Initialized, Role: FOLLOWER
I20260806 23:52:07.942528 18265 raft_consensus.cc:348] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } }
I20260806 23:52:07.942641 18269 consensus_queue.cc:260] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } }
I20260806 23:52:07.942759 18265 raft_consensus.cc:374] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:07.942813 18265 raft_consensus.cc:729] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a445f91c5a343778021e10686572c61, State: Initialized, Role: FOLLOWER
I20260806 23:52:07.942814 18263 consensus_queue.cc:260] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } }
I20260806 23:52:07.943195 18265 consensus_queue.cc:260] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } }
I20260806 23:52:07.943588 18240 heartbeater.cc:503] Master 127.10.225.190:39423 was elected leader, sending a full tablet report...
I20260806 23:52:07.943853 18263 ts_tablet_manager.cc:1428] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:52:07.944072 18265 ts_tablet_manager.cc:1428] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:52:07.944407 17830 heartbeater.cc:503] Master 127.10.225.190:39423 was elected leader, sending a full tablet report...
I20260806 23:52:07.944432 18269 ts_tablet_manager.cc:1428] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:52:07.944741 18035 heartbeater.cc:503] Master 127.10.225.190:39423 was elected leader, sending a full tablet report...
I20260806 23:52:08.012513 18275 raft_consensus.cc:482] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:08.012862 18275 raft_consensus.cc:504] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } }
I20260806 23:52:08.014114 18275 leader_election.cc:288] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34a402917de0493781fa5b49bd3cdd57 (127.10.225.131:34713), 9a445f91c5a343778021e10686572c61 (127.10.225.129:37803)
I20260806 23:52:08.017779 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.014421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:37803 (local address 127.10.225.130:34667)
0806 23:52:08.014540 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:08.014555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:08.014656 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:08.014969 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:08.014972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:08.014991 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:08.015114 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:08.015119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.015897 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.015900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:08.016694 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:08.016703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.017364 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.017367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:08.017476 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:08.017495 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 23:52:08.017560 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:52:08.017635 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"spinlock_wait_cycles":3328}
I20260806 23:52:08.017779 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.014586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:34667 (local address 127.10.225.129:37803)
0806 23:52:08.014716 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:08.014721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:08.014736 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:52:08.014795 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:08.014797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:08.014855 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:08.014933 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:08.015220 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.015786 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.016815 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.017240 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.017354 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:08.017413 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:08.017586 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 23:52:08.017665 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:52:08.018141 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.014598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:45249 (local address 127.10.225.131:34713)
0806 23:52:08.014716 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:08.014721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:08.015010 (+ 289us) server_negotiation.cc:407] Connection header received
0806 23:52:08.015141 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:08.015144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:08.015187 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:08.015249 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:08.015510 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.016044 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.017188 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.017590 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.017695 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:08.017746 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:08.017946 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 23:52:08.018026 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:52:08.018476 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.014424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:34713 (local address 127.10.225.130:45249)
0806 23:52:08.014915 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:08.014926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:08.015040 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:08.015272 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:08.015275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:08.015290 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:08.015409 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:08.015415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.016148 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.016151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:08.017031 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:08.017042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.017705 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.017709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:08.017803 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:08.017823 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 23:52:08.018296 (+ 473us) client_negotiation.cc:240] Negotiation successful
0806 23:52:08.018366 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":131,"spinlock_wait_cycles":2304,"thread_start_us":146,"threads_started":1}
I20260806 23:52:08.018491 17754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4918140c47e3a4bdc07ddbd22456" candidate_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a445f91c5a343778021e10686572c61" is_pre_election: true
I20260806 23:52:08.018839 17754 raft_consensus.cc:2454] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc2b9d12165499f93eec5cc1a432f4a in term 0.
I20260806 23:52:08.019049 18168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4918140c47e3a4bdc07ddbd22456" candidate_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a402917de0493781fa5b49bd3cdd57" is_pre_election: true
I20260806 23:52:08.019304 18168 raft_consensus.cc:2454] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc2b9d12165499f93eec5cc1a432f4a in term 0.
I20260806 23:52:08.019362 17866 leader_election.cc:302] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a445f91c5a343778021e10686572c61, bfc2b9d12165499f93eec5cc1a432f4a; no voters:
I20260806 23:52:08.019632 18275 raft_consensus.cc:2781] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:08.019749 18275 raft_consensus.cc:482] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:08.019814 18275 raft_consensus.cc:3037] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:08.020892 18275 raft_consensus.cc:504] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } }
I20260806 23:52:08.021234 18275 leader_election.cc:288] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [CANDIDATE]: Term 1 election: Requested vote from peers 34a402917de0493781fa5b49bd3cdd57 (127.10.225.131:34713), 9a445f91c5a343778021e10686572c61 (127.10.225.129:37803)
I20260806 23:52:08.021634 18168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4918140c47e3a4bdc07ddbd22456" candidate_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a402917de0493781fa5b49bd3cdd57"
I20260806 23:52:08.021667 17754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfd4918140c47e3a4bdc07ddbd22456" candidate_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a445f91c5a343778021e10686572c61"
I20260806 23:52:08.021745 18168 raft_consensus.cc:3037] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:08.021790 17754 raft_consensus.cc:3037] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:08.022820 18168 raft_consensus.cc:2454] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc2b9d12165499f93eec5cc1a432f4a in term 1.
I20260806 23:52:08.022900 17754 raft_consensus.cc:2454] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc2b9d12165499f93eec5cc1a432f4a in term 1.
I20260806 23:52:08.023128 17861 leader_election.cc:302] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a402917de0493781fa5b49bd3cdd57, bfc2b9d12165499f93eec5cc1a432f4a; no voters:
I20260806 23:52:08.023303 18275 raft_consensus.cc:2781] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:08.023554 18275 raft_consensus.cc:686] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 1 LEADER]: Becoming Leader. State: Replica: bfc2b9d12165499f93eec5cc1a432f4a, State: Running, Role: LEADER
I20260806 23:52:08.023911 18275 consensus_queue.cc:237] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } }
I20260806 23:52:08.027323 17515 catalog_manager.cc:5495] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a reported cstate change: term changed from 0 to 1, leader changed from <none> to bfc2b9d12165499f93eec5cc1a432f4a (127.10.225.130). New cstate: current_term: 1 leader_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfc2b9d12165499f93eec5cc1a432f4a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 43137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 } health_report { overall_health: UNKNOWN } } }
I20260806 23:52:08.045177 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.041424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.10.225.130:43137)
0806 23:52:08.041560 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:08.041565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:08.041579 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:08.041662 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:08.041664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:08.041721 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:08.041815 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:08.042169 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.042693 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.043539 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:08.043782 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:08.043870 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:08.044310 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:08.044773 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:08.044882 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:08.044964 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 23:52:08.045029 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:52:08.053205 18168 raft_consensus.cc:1261] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [term 1 FOLLOWER]: Refusing update from remote peer bfc2b9d12165499f93eec5cc1a432f4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:08.053205 17754 raft_consensus.cc:1261] T ccfd4918140c47e3a4bdc07ddbd22456 P 9a445f91c5a343778021e10686572c61 [term 1 FOLLOWER]: Refusing update from remote peer bfc2b9d12165499f93eec5cc1a432f4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:08.053982 18275 consensus_queue.cc:1035] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a445f91c5a343778021e10686572c61" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:08.054198 18282 consensus_queue.cc:1035] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a402917de0493781fa5b49bd3cdd57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:08.064698 18289 mvcc.cc:205] Tried to move back new op lower bound from 7315703103696207872 to 7315703103588655104. Current Snapshot: MvccSnapshot[applied={T|T < 7315703103696207872}]
I20260806 23:52:08.067785 18286 mvcc.cc:205] Tried to move back new op lower bound from 7315703103696207872 to 7315703103588655104. Current Snapshot: MvccSnapshot[applied={T|T < 7315703103696207872}]
I20260806 23:52:08.077418 17521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:08.077613 17521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:08.080157 17521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9af0b6ee7794fb7a8c6400ccdef6b27: Sending DeleteTablet for 3 replicas of tablet ccfd4918140c47e3a4bdc07ddbd22456
I20260806 23:52:08.080999 17935 tablet_service.cc:1514] Processing DeleteTablet for tablet ccfd4918140c47e3a4bdc07ddbd22456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:08 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:39276
I20260806 23:52:08.081172 18141 tablet_service.cc:1514] Processing DeleteTablet for tablet ccfd4918140c47e3a4bdc07ddbd22456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:08 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:41572
I20260806 23:52:08.081154 17722 tablet_service.cc:1514] Processing DeleteTablet for tablet ccfd4918140c47e3a4bdc07ddbd22456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:08 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:47688
I20260806 23:52:08.081748 18297 tablet_replica.cc:330] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a: stopping tablet replica
I20260806 23:52:08.082002 18298 tablet_replica.cc:330] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: stopping tablet replica
I20260806 23:52:08.082209 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 17616
I20260806 23:52:08.082330 18298 raft_consensus.cc:2229] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:08.082691 18298 raft_consensus.cc:2258] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:08.089099 18297 raft_consensus.cc:2229] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:52:08.088747 18298 ts_tablet_manager.cc:1905] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:08.092869 18298 ts_tablet_manager.cc:1918] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:08.093011 18298 log.cc:1196] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/ccfd4918140c47e3a4bdc07ddbd22456
I20260806 23:52:08.093547 18297 raft_consensus.cc:2258] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:08.094149 18298 ts_tablet_manager.cc:1939] T ccfd4918140c47e3a4bdc07ddbd22456 P 34a402917de0493781fa5b49bd3cdd57: Deleting consensus metadata
I20260806 23:52:08.095526 18297 ts_tablet_manager.cc:1905] T ccfd4918140c47e3a4bdc07ddbd22456 P bfc2b9d12165499f93eec5cc1a432f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 23:52:08.095856 17506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:37803 (error 108)
I20260806 23:52:08.095976 17500 catalog_manager.cc:4841] TS 34a402917de0493781fa5b49bd3cdd57 (127.10.225.131:34713): tablet ccfd4918140c47e3a4bdc07ddbd22456 (table test-table [id=8502440e721f43b6b17e4732c5186e0f]) successfully deleted
W20260806 23:52:08.097012 18258 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:08.096648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:37803 (local address 127.0.0.1:47700)
0806 23:52:08.096837 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:08.096911 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:37803: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":109}
I20260806 23:52:08.097090 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 17841
W20260806 23:52:08.099925 17506 catalog_manager.cc:4584] TS 9a445f91c5a343778021e10686572c61 (127.10.225.129:37803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccfd4918140c47e3a4bdc07ddbd22456: Network error: Client connection negotiation failed: client connection to 127.10.225.129:37803: connect: Connection reset by peer (error 104)
I20260806 23:52:08.108911 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 18041
I20260806 23:52:08.118610 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.109159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:43137 (local address 127.0.0.1:39288)
0806 23:52:08.118376 (+ 9217us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:08.118486 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.130:43137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9054}
I20260806 23:52:08.125849 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 17475
2026-08-06T23:52:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2585 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T23:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:08Z Disabled control of system clock
I20260806 23:52:08.166101 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:43475
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 23:52:08.289405 18308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:08.289734 18308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:08.289772 18308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:08.293917 18308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:08.293996 18308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:08.294011 18308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:08.294027 18308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:08.294041 18308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:08.299093 18308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:43475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:08.300300 18308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:08.301723 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:08.310591 18314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:08.310600 18317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:08.310644 18308 server_base.cc:1034] running on GCE node
W20260806 23:52:08.310591 18315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:08.311408 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:08.312611 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:08.313822 18308 hybrid_clock.cc:648] HybridClock initialized: now 1786060328313803 us; error 34 us; skew 500 ppm
I20260806 23:52:08.315922 18308 webserver.cc:460] Webserver started at http://127.10.225.190:41303/ using document root <none> and password file <none>
I20260806 23:52:08.316522 18308 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:08.316588 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:08.316797 18308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:08.318957 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "b2f9860c93b14aa1a509bd032a1b1ca7"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "0dff07a206903ce2493209b214a4160a"
server_key_iv: "9f3cb5934963944ff4fde5e3486ef23b"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.319520 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "b2f9860c93b14aa1a509bd032a1b1ca7"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "0dff07a206903ce2493209b214a4160a"
server_key_iv: "9f3cb5934963944ff4fde5e3486ef23b"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.323819 18308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 23:52:08.327119 18331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:08.328745 18308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:52:08.328903 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "b2f9860c93b14aa1a509bd032a1b1ca7"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "0dff07a206903ce2493209b214a4160a"
server_key_iv: "9f3cb5934963944ff4fde5e3486ef23b"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.329030 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:08.333935 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:08.334584 18308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:08.334757 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:08.341786 18308 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:43475
I20260806 23:52:08.341802 18415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:43475 every 8 connection(s)
I20260806 23:52:08.342760 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:08.344235 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 18308
I20260806 23:52:08.344484 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:08.344992 11142 external_mini_cluster.cc:1428] Setting key 27d52d882cba16c8631823983e8e3c20
I20260806 23:52:08.346779 18416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:08.351840 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.346667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.10.225.190:43475)
0806 23:52:08.347406 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:08.347426 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:08.347470 (+ 44us) server_negotiation.cc:407] Connection header received
0806 23:52:08.348603 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:08.348627 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:08.349094 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:08.349486 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:08.350245 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:08.350251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:08.350265 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:08.350278 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:08.350355 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:08.350378 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:08.350526 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:08.350596 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 23:52:08.350870 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":131,"threads_started":1}
I20260806 23:52:08.354234 18416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7: Bootstrap starting.
I20260806 23:52:08.356501 18416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:08.357193 18416 log.cc:824] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:08.359200 18416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7: No bootstrap required, opened a new log
I20260806 23:52:08.362914 18416 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f9860c93b14aa1a509bd032a1b1ca7" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43475 } }
I20260806 23:52:08.363149 18416 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:08.363183 18416 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f9860c93b14aa1a509bd032a1b1ca7, State: Initialized, Role: FOLLOWER
I20260806 23:52:08.363595 18416 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f9860c93b14aa1a509bd032a1b1ca7" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43475 } }
I20260806 23:52:08.363761 18416 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:08.363835 18416 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:08.363986 18416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:08.364949 18416 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f9860c93b14aa1a509bd032a1b1ca7" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43475 } }
I20260806 23:52:08.365276 18416 leader_election.cc:302] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f9860c93b14aa1a509bd032a1b1ca7; no voters:
I20260806 23:52:08.365603 18416 leader_election.cc:288] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:08.365732 18423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:08.365969 18423 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [term 1 LEADER]: Becoming Leader. State: Replica: b2f9860c93b14aa1a509bd032a1b1ca7, State: Running, Role: LEADER
I20260806 23:52:08.366345 18423 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f9860c93b14aa1a509bd032a1b1ca7" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43475 } }
I20260806 23:52:08.367049 18416 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:08.368214 18424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f9860c93b14aa1a509bd032a1b1ca7. Latest consensus state: current_term: 1 leader_uuid: "b2f9860c93b14aa1a509bd032a1b1ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f9860c93b14aa1a509bd032a1b1ca7" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43475 } } }
I20260806 23:52:08.368216 18426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2f9860c93b14aa1a509bd032a1b1ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f9860c93b14aa1a509bd032a1b1ca7" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43475 } } }
I20260806 23:52:08.368359 18424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:08.368372 18426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:08.369313 18437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:08.372262 18437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:08.378185 18437 catalog_manager.cc:1347] Generated new cluster ID: 672b27dc388643f9823e40ef45dec067
I20260806 23:52:08.378258 18437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:08.394316 18437 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:08.394687 18437 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:08.404038 18437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7: Generated new TSK 0
I20260806 23:52:08.404841 18437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:52:08.407930 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43475
--builtin_ntp_servers=127.10.225.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 23:52:08.549818 18458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:08.550143 18458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:08.550189 18458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:08.554839 18458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:08.554922 18458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:08.554984 18458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:08.561110 18458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:08.562367 18458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:08.563906 18458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:08.574694 18465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:08.574796 18467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:08.574813 18469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:08.575217 18458 server_base.cc:1034] running on GCE node
I20260806 23:52:08.575721 18458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:08.576414 18458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:08.577597 18458 hybrid_clock.cc:648] HybridClock initialized: now 1786060328577558 us; error 48 us; skew 500 ppm
I20260806 23:52:08.579550 18458 webserver.cc:460] Webserver started at http://127.10.225.129:44993/ using document root <none> and password file <none>
I20260806 23:52:08.580165 18458 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:08.580230 18458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:08.580433 18458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:08.582556 18458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "79958e0ed478424a98f35a1c1fa997d9"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "84b3db53b719267cb17b73f65bf15782"
server_key_iv: "81ab139d981a222bd89c6b80e2e4ba61"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.583132 18458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "79958e0ed478424a98f35a1c1fa997d9"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "84b3db53b719267cb17b73f65bf15782"
server_key_iv: "81ab139d981a222bd89c6b80e2e4ba61"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.587301 18458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:08.590160 18478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:08.591720 18458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 23:52:08.591858 18458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "79958e0ed478424a98f35a1c1fa997d9"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "84b3db53b719267cb17b73f65bf15782"
server_key_iv: "81ab139d981a222bd89c6b80e2e4ba61"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.591972 18458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:08.597277 18458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:08.597934 18458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:08.598104 18458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:08.598665 18458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:08.599646 18458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:08.599723 18458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:08.599781 18458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:08.599805 18458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:08.609043 18458 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:41797
I20260806 23:52:08.609122 18640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:41797 every 8 connection(s)
I20260806 23:52:08.610086 18458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:08.616917 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 18458
I20260806 23:52:08.617058 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:08.617401 11142 external_mini_cluster.cc:1428] Setting key ae99f1799d330c569b5159dc71db7da8
I20260806 23:52:08.618361 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.612111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:40385 (local address 127.10.225.190:43475)
0806 23:52:08.612387 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:08.612392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:08.613542 (+ 1150us) server_negotiation.cc:407] Connection header received
0806 23:52:08.615206 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:08.615209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:08.615264 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:08.615343 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:08.617819 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:08.617823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:08.617825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:08.617839 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:08.617863 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:08.617871 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:08.617963 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:08.618171 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 23:52:08.618226 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 23:52:08.619310 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.612536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:43475 (local address 127.10.225.129:40385)
0806 23:52:08.613364 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:08.613408 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:08.614979 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:08.615560 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:08.615569 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:08.615702 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:08.616394 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:08.616396 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:08.616451 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:08.616544 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:08.617962 (+ 1418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:08.617974 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:08.617981 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 23:52:08.618169 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 23:52:08.618409 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":114,"threads_started":1}
I20260806 23:52:08.619565 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43475
--builtin_ntp_servers=127.10.225.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 23:52:08.620652 18645 heartbeater.cc:347] Connected to a master server at 127.10.225.190:43475
I20260806 23:52:08.620893 18645 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:08.621323 18645 heartbeater.cc:511] Master 127.10.225.190:43475 requested a full tablet report, sending...
I20260806 23:52:08.623929 18362 ts_manager.cc:194] Registered new tserver with Master: 79958e0ed478424a98f35a1c1fa997d9 (127.10.225.129:41797)
W20260806 23:52:08.749601 18649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:08.749931 18649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:08.749975 18649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:08.754482 18649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:08.754563 18649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:08.754626 18649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:08.759953 18649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:08.761173 18649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:08.762598 18649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:08.774156 18666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:08.774241 18663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:08.774262 18660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:08.774608 18649 server_base.cc:1034] running on GCE node
I20260806 23:52:08.775068 18649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:08.775849 18649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:08.777064 18649 hybrid_clock.cc:648] HybridClock initialized: now 1786060328777043 us; error 49 us; skew 500 ppm
I20260806 23:52:08.779906 18649 webserver.cc:460] Webserver started at http://127.10.225.130:34927/ using document root <none> and password file <none>
I20260806 23:52:08.780562 18649 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:08.780637 18649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:08.780857 18649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:08.783073 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "13d183295820422888516cabcc71ddc2"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "c2c5d8d150e7095d624034e5f47d8043"
server_key_iv: "28573f950d2e6025d16028b7e49c3484"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.783712 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "13d183295820422888516cabcc71ddc2"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "c2c5d8d150e7095d624034e5f47d8043"
server_key_iv: "28573f950d2e6025d16028b7e49c3484"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.789269 18649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 23:52:08.792816 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:08.794713 18649 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 23:52:08.794866 18649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "13d183295820422888516cabcc71ddc2"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "c2c5d8d150e7095d624034e5f47d8043"
server_key_iv: "28573f950d2e6025d16028b7e49c3484"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.794997 18649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:08.800242 18649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:08.801014 18649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:08.801210 18649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:08.801795 18649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:08.802889 18649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:08.802948 18649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:08.803006 18649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:08.803036 18649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:08.814374 18649 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:41019
I20260806 23:52:08.814462 18841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:41019 every 8 connection(s)
I20260806 23:52:08.815358 18649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:08.819764 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 18649
I20260806 23:52:08.819908 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:08.820216 11142 external_mini_cluster.cc:1428] Setting key e8eff2fb7acd2377486a1ecfde57aa69
I20260806 23:52:08.823833 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.818279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:44849 (local address 127.10.225.190:43475)
0806 23:52:08.818461 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:08.818465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:08.819719 (+ 1254us) server_negotiation.cc:407] Connection header received
0806 23:52:08.821442 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:08.821447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:08.821559 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:08.821695 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:08.822921 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:08.822925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:08.822928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:08.822941 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:08.822967 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:08.822977 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:08.823079 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:08.823259 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 23:52:08.823429 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 23:52:08.823876 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43475
--builtin_ntp_servers=127.10.225.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 23:52:08.824602 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:08.818671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:43475 (local address 127.10.225.130:44849)
0806 23:52:08.819492 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:08.819550 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:08.821121 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:08.821910 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:08.821917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:08.822023 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:08.822620 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:08.822622 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:08.822667 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:08.822744 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:08.823104 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:08.823114 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:08.823120 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:52:08.823305 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 23:52:08.823611 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":150,"threads_started":1}
I20260806 23:52:08.825889 18843 heartbeater.cc:347] Connected to a master server at 127.10.225.190:43475
I20260806 23:52:08.826143 18843 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:08.826637 18843 heartbeater.cc:511] Master 127.10.225.190:43475 requested a full tablet report, sending...
I20260806 23:52:08.827811 18362 ts_manager.cc:194] Registered new tserver with Master: 13d183295820422888516cabcc71ddc2 (127.10.225.130:41019)
W20260806 23:52:08.955973 18848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:08.956435 18848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:08.956593 18848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:08.963152 18848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:08.963264 18848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:08.963343 18848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:08.969857 18848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:08.971098 18848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:08.972651 18848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:08.983418 18855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:08.983513 18848 server_base.cc:1034] running on GCE node
W20260806 23:52:08.983445 18857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:08.983418 18859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:08.984145 18848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:08.984850 18848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:08.986028 18848 hybrid_clock.cc:648] HybridClock initialized: now 1786060328985964 us; error 74 us; skew 500 ppm
I20260806 23:52:08.988056 18848 webserver.cc:460] Webserver started at http://127.10.225.131:46321/ using document root <none> and password file <none>
I20260806 23:52:08.988675 18848 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:08.988750 18848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:08.988981 18848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:08.991263 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "b550b0a6d3474992905775e5f730c524"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "95c858474d55a25286e62076af7a91b4"
server_key_iv: "21a6fcafd43755fad309528653e63c92"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.991915 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "b550b0a6d3474992905775e5f730c524"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "95c858474d55a25286e62076af7a91b4"
server_key_iv: "21a6fcafd43755fad309528653e63c92"
server_key_version: "encryptionkey@0"
I20260806 23:52:08.996116 18848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:52:08.999079 18868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:09.000964 18848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:09.001217 18848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "b550b0a6d3474992905775e5f730c524"
format_stamp: "Formatted at 2026-08-06 23:52:08 on dist-test-slave-nj21"
server_key: "95c858474d55a25286e62076af7a91b4"
server_key_iv: "21a6fcafd43755fad309528653e63c92"
server_key_version: "encryptionkey@0"
I20260806 23:52:09.001371 18848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:09.007639 18848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:09.008426 18848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:09.008599 18848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:09.009192 18848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:09.010223 18848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:09.010275 18848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:09.010334 18848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:09.010358 18848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:09.019639 18848 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:35309
I20260806 23:52:09.019744 19043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:35309 every 8 connection(s)
I20260806 23:52:09.020644 18848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:09.023075 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 18848
I20260806 23:52:09.023195 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:09.023557 11142 external_mini_cluster.cc:1428] Setting key bfe2726d677f8878accc0a5c8550bb9e
I20260806 23:52:09.028752 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.022853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:54863 (local address 127.10.225.190:43475)
0806 23:52:09.023130 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.023136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.025029 (+ 1893us) server_negotiation.cc:407] Connection header received
0806 23:52:09.026894 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.026897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.026943 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.027013 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.028270 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.028274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.028276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.028289 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.028309 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.028317 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.028406 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.028580 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.028630 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260806 23:52:09.029755 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.023576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:43475 (local address 127.10.225.131:54863)
0806 23:52:09.024790 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:09.024861 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:09.026636 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:09.027191 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:09.027200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:09.027311 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:09.027977 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:09.027979 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:09.028026 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:09.028108 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:09.028431 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:09.028440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:09.028446 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:52:09.028629 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 23:52:09.028891 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":281,"threads_started":1}
I20260806 23:52:09.031018 19044 heartbeater.cc:347] Connected to a master server at 127.10.225.190:43475
I20260806 23:52:09.031253 19044 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:09.031735 19044 heartbeater.cc:511] Master 127.10.225.190:43475 requested a full tablet report, sending...
I20260806 23:52:09.032866 18362 ts_manager.cc:194] Registered new tserver with Master: b550b0a6d3474992905775e5f730c524 (127.10.225.131:35309)
I20260806 23:52:09.039016 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:52:09.042790 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.041117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.10.225.190:43475)
0806 23:52:09.041241 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.041245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.041355 (+ 110us) server_negotiation.cc:407] Connection header received
0806 23:52:09.041439 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.041441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.041479 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.041549 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.042237 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.042240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.042242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.042253 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.042275 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.042283 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.042353 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.042646 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.042681 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:52:09.045746 18362 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:52:09.049108 18362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:09.061730 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.058563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:41019 (local address 127.0.0.1:49924)
0806 23:52:09.059634 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:09.059645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:09.059746 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:09.060301 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:09.060307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:09.060387 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:09.060948 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:09.060951 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:09.060971 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:09.061043 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:09.061388 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:09.061397 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:09.061403 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:52:09.061508 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 23:52:09.061578 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":111,"threads_started":1}
I20260806 23:52:09.061749 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.058915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.10.225.130:41019)
0806 23:52:09.059814 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.059829 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.059871 (+ 42us) server_negotiation.cc:407] Connection header received
0806 23:52:09.059929 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.059935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.060137 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.060656 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.061210 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.061216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.061228 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.061239 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.061281 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.061306 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.061408 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.061532 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.061604 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":86,"threads_started":1}
I20260806 23:52:09.062129 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.059340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:41797 (local address 127.0.0.1:54146)
0806 23:52:09.059761 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:09.059775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:09.059858 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:09.061057 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:09.061060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:09.061075 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:09.061459 (+ 384us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:09.061462 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:09.061473 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:09.061503 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:09.061889 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:09.061892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:09.061895 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 23:52:09.061961 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 23:52:09.061995 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":89,"threads_started":1}
I20260806 23:52:09.062137 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.058554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:35309 (local address 127.0.0.1:41698)
0806 23:52:09.059450 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:09.059500 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:09.060101 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:09.060513 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:09.060516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:09.060529 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:09.061178 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:09.061179 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:09.061188 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:09.061208 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:09.061953 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:09.061956 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:09.061959 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 23:52:09.062016 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 23:52:09.062040 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"spinlock_wait_cycles":10624,"thread_start_us":105,"threads_started":1}
I20260806 23:52:09.062417 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.058745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.10.225.131:35309)
0806 23:52:09.059427 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.059440 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.059957 (+ 517us) server_negotiation.cc:407] Connection header received
0806 23:52:09.060230 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.060236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.060387 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.060506 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.061749 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.061754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.061763 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.061773 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.061807 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.061830 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.061918 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.062197 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.062270 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":114,"threads_started":1}
I20260806 23:52:09.062840 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.059515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.10.225.129:41797)
0806 23:52:09.059985 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.059996 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.060019 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:52:09.060073 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.060078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.060261 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.060390 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.061645 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.061651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.061662 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.061674 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.061715 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.061743 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.061844 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.062482 (+ 638us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.062551 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":113,"threads_started":1}
I20260806 23:52:09.064921 18748 tablet_service.cc:1467] Processing CreateTablet for tablet 3937f3e1b5ab43a9a40d312174a001a8 (DEFAULT_TABLE table=test-table [id=5fcd74f5be4e40dc87ebda0cdcd1d4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:09.065522 18949 tablet_service.cc:1467] Processing CreateTablet for tablet 3937f3e1b5ab43a9a40d312174a001a8 (DEFAULT_TABLE table=test-table [id=5fcd74f5be4e40dc87ebda0cdcd1d4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:09.065523 18549 tablet_service.cc:1467] Processing CreateTablet for tablet 3937f3e1b5ab43a9a40d312174a001a8 (DEFAULT_TABLE table=test-table [id=5fcd74f5be4e40dc87ebda0cdcd1d4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:09.066001 18748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3937f3e1b5ab43a9a40d312174a001a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:09.066509 18949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3937f3e1b5ab43a9a40d312174a001a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:09.066509 18549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3937f3e1b5ab43a9a40d312174a001a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:09.073580 19070 tablet_bootstrap.cc:492] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: Bootstrap starting.
I20260806 23:52:09.074625 19072 tablet_bootstrap.cc:492] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9: Bootstrap starting.
I20260806 23:52:09.076025 19075 tablet_bootstrap.cc:492] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: Bootstrap starting.
I20260806 23:52:09.076759 19070 tablet_bootstrap.cc:654] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:09.076881 19072 tablet_bootstrap.cc:654] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:09.077651 19070 log.cc:824] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:09.077718 19072 log.cc:824] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:09.078150 19075 tablet_bootstrap.cc:654] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:09.078814 19075 log.cc:824] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:09.079564 19070 tablet_bootstrap.cc:492] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: No bootstrap required, opened a new log
I20260806 23:52:09.079792 19070 ts_tablet_manager.cc:1397] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:52:09.080554 19075 tablet_bootstrap.cc:492] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: No bootstrap required, opened a new log
I20260806 23:52:09.080737 19075 ts_tablet_manager.cc:1397] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:52:09.080976 19072 tablet_bootstrap.cc:492] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9: No bootstrap required, opened a new log
I20260806 23:52:09.081163 19072 ts_tablet_manager.cc:1397] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 23:52:09.084360 19075 raft_consensus.cc:348] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } }
I20260806 23:52:09.084589 19075 raft_consensus.cc:374] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:09.084641 19075 raft_consensus.cc:729] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b550b0a6d3474992905775e5f730c524, State: Initialized, Role: FOLLOWER
I20260806 23:52:09.084513 19070 raft_consensus.cc:348] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } }
I20260806 23:52:09.084753 19070 raft_consensus.cc:374] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:09.084800 19070 raft_consensus.cc:729] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d183295820422888516cabcc71ddc2, State: Initialized, Role: FOLLOWER
I20260806 23:52:09.085132 19075 consensus_queue.cc:260] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } }
I20260806 23:52:09.085209 19070 consensus_queue.cc:260] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } }
I20260806 23:52:09.085474 19072 raft_consensus.cc:348] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } }
I20260806 23:52:09.085724 19072 raft_consensus.cc:374] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:09.085778 19072 raft_consensus.cc:729] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79958e0ed478424a98f35a1c1fa997d9, State: Initialized, Role: FOLLOWER
I20260806 23:52:09.085839 19044 heartbeater.cc:503] Master 127.10.225.190:43475 was elected leader, sending a full tablet report...
I20260806 23:52:09.086189 19075 ts_tablet_manager.cc:1428] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 23:52:09.086169 19072 consensus_queue.cc:260] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } }
I20260806 23:52:09.086766 18645 heartbeater.cc:503] Master 127.10.225.190:43475 was elected leader, sending a full tablet report...
I20260806 23:52:09.087080 19072 ts_tablet_manager.cc:1428] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:52:09.087858 19070 ts_tablet_manager.cc:1428] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 23:52:09.088172 18843 heartbeater.cc:503] Master 127.10.225.190:43475 was elected leader, sending a full tablet report...
W20260806 23:52:09.111478 18647 tablet.cc:2369] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:52:09.184957 19081 raft_consensus.cc:482] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:09.185285 19081 raft_consensus.cc:504] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } }
I20260806 23:52:09.187086 19081 leader_election.cc:288] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b550b0a6d3474992905775e5f730c524 (127.10.225.131:35309), 79958e0ed478424a98f35a1c1fa997d9 (127.10.225.129:41797)
I20260806 23:52:09.189121 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.187384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:35309 (local address 127.10.225.130:44329)
0806 23:52:09.187510 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:09.187526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:09.187614 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:09.187927 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:09.187932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:09.187950 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:09.188516 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:09.188518 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:09.188531 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:09.188560 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:09.188855 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:09.188859 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:09.188863 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:52:09.188953 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 23:52:09.188989 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 23:52:09.189136 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.187477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:44329 (local address 127.10.225.131:35309)
0806 23:52:09.187595 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.187599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.187614 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:52:09.187739 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.187743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.187786 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.187864 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.188689 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.188693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.188696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.188708 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.188729 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.188737 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.188819 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.188963 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.189009 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:52:09.189630 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.187383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:41797 (local address 127.10.225.130:48667)
0806 23:52:09.188069 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:09.188082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:09.188170 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:09.188431 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:09.188434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:09.188449 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:09.188903 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:09.188904 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:09.188913 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:09.188938 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:09.189309 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:09.189312 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:09.189316 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:52:09.189441 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 23:52:09.189489 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":319,"thread_start_us":159,"threads_started":1}
I20260806 23:52:09.189755 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.187562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:48667 (local address 127.10.225.129:41797)
0806 23:52:09.187724 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.187727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.188183 (+ 456us) server_negotiation.cc:407] Connection header received
0806 23:52:09.188243 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.188246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.188309 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.188405 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.189066 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.189070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.189072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.189084 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.189104 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.189113 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.189204 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.189469 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.189512 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:52:09.190138 18978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3937f3e1b5ab43a9a40d312174a001a8" candidate_uuid: "13d183295820422888516cabcc71ddc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550b0a6d3474992905775e5f730c524" is_pre_election: true
I20260806 23:52:09.190564 18978 raft_consensus.cc:2454] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d183295820422888516cabcc71ddc2 in term 0.
I20260806 23:52:09.190943 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3937f3e1b5ab43a9a40d312174a001a8" candidate_uuid: "13d183295820422888516cabcc71ddc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79958e0ed478424a98f35a1c1fa997d9" is_pre_election: true
I20260806 23:52:09.191077 18683 leader_election.cc:302] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d183295820422888516cabcc71ddc2, b550b0a6d3474992905775e5f730c524; no voters:
I20260806 23:52:09.191180 18580 raft_consensus.cc:2454] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d183295820422888516cabcc71ddc2 in term 0.
I20260806 23:52:09.191377 19081 raft_consensus.cc:2781] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:09.191476 19081 raft_consensus.cc:482] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:09.191532 19081 raft_consensus.cc:3037] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:09.192677 19081 raft_consensus.cc:504] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } }
I20260806 23:52:09.193023 19081 leader_election.cc:288] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [CANDIDATE]: Term 1 election: Requested vote from peers b550b0a6d3474992905775e5f730c524 (127.10.225.131:35309), 79958e0ed478424a98f35a1c1fa997d9 (127.10.225.129:41797)
I20260806 23:52:09.193403 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3937f3e1b5ab43a9a40d312174a001a8" candidate_uuid: "13d183295820422888516cabcc71ddc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79958e0ed478424a98f35a1c1fa997d9"
I20260806 23:52:09.193526 18580 raft_consensus.cc:3037] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:09.193579 18978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3937f3e1b5ab43a9a40d312174a001a8" candidate_uuid: "13d183295820422888516cabcc71ddc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b550b0a6d3474992905775e5f730c524"
I20260806 23:52:09.193706 18978 raft_consensus.cc:3037] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:09.194842 18580 raft_consensus.cc:2454] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d183295820422888516cabcc71ddc2 in term 1.
I20260806 23:52:09.194907 18978 raft_consensus.cc:2454] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d183295820422888516cabcc71ddc2 in term 1.
I20260806 23:52:09.195185 18676 leader_election.cc:302] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d183295820422888516cabcc71ddc2, 79958e0ed478424a98f35a1c1fa997d9; no voters:
I20260806 23:52:09.195473 19081 raft_consensus.cc:2781] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:09.195883 19081 raft_consensus.cc:686] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 1 LEADER]: Becoming Leader. State: Replica: 13d183295820422888516cabcc71ddc2, State: Running, Role: LEADER
I20260806 23:52:09.196362 19081 consensus_queue.cc:237] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } }
I20260806 23:52:09.200326 18362 catalog_manager.cc:5495] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d183295820422888516cabcc71ddc2 (127.10.225.130). New cstate: current_term: 1 leader_uuid: "13d183295820422888516cabcc71ddc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d183295820422888516cabcc71ddc2" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 } health_report { overall_health: UNKNOWN } } }
I20260806 23:52:09.219730 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.218163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.10.225.130:41019)
0806 23:52:09.218296 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.218300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.218316 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:52:09.218448 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.218451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.218520 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.218593 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.219312 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.219315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.219317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.219328 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.219347 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.219356 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.219436 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.219518 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.219561 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:52:09.227177 18978 raft_consensus.cc:1261] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [term 1 FOLLOWER]: Refusing update from remote peer 13d183295820422888516cabcc71ddc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:09.227177 18580 raft_consensus.cc:1261] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9 [term 1 FOLLOWER]: Refusing update from remote peer 13d183295820422888516cabcc71ddc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:09.227943 19086 consensus_queue.cc:1035] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79958e0ed478424a98f35a1c1fa997d9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:09.228196 19081 consensus_queue.cc:1035] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b550b0a6d3474992905775e5f730c524" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:09.237421 19094 mvcc.cc:205] Tried to move back new op lower bound from 7315703108505600000 to 7315703108392058880. Current Snapshot: MvccSnapshot[applied={T|T < 7315703108505600000}]
I20260806 23:52:09.249070 18358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:09.249257 18358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:09.251748 18358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2f9860c93b14aa1a509bd032a1b1ca7: Sending DeleteTablet for 3 replicas of tablet 3937f3e1b5ab43a9a40d312174a001a8
I20260806 23:52:09.252607 18949 tablet_service.cc:1514] Processing DeleteTablet for tablet 3937f3e1b5ab43a9a40d312174a001a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:09 UTC) from {username='slave'} at 127.0.0.1:41698
I20260806 23:52:09.252767 18748 tablet_service.cc:1514] Processing DeleteTablet for tablet 3937f3e1b5ab43a9a40d312174a001a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:09 UTC) from {username='slave'} at 127.0.0.1:49924
I20260806 23:52:09.253180 19105 tablet_replica.cc:330] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: stopping tablet replica
I20260806 23:52:09.253190 18549 tablet_service.cc:1514] Processing DeleteTablet for tablet 3937f3e1b5ab43a9a40d312174a001a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:09 UTC) from {username='slave'} at 127.0.0.1:54146
I20260806 23:52:09.253456 19105 raft_consensus.cc:2229] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:09.253677 19108 tablet_replica.cc:330] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9: stopping tablet replica
I20260806 23:52:09.253785 19105 raft_consensus.cc:2258] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:09.253947 19108 raft_consensus.cc:2229] T 3937f3e1b5ab43a9a40d312174a001a8 P 79958e0ed478424a98f35a1c1fa997d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:09.254069 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 18458
I20260806 23:52:09.259555 19107 tablet_replica.cc:330] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: stopping tablet replica
I20260806 23:52:09.259900 19107 raft_consensus.cc:2229] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:52:09.260344 19107 raft_consensus.cc:2258] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:09.260411 19105 ts_tablet_manager.cc:1905] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:09.261798 19107 ts_tablet_manager.cc:1905] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:09.263473 19105 ts_tablet_manager.cc:1918] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:09.263589 19105 log.cc:1196] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/3937f3e1b5ab43a9a40d312174a001a8
I20260806 23:52:09.263967 19105 ts_tablet_manager.cc:1939] T 3937f3e1b5ab43a9a40d312174a001a8 P b550b0a6d3474992905775e5f730c524: Deleting consensus metadata
I20260806 23:52:09.264799 19107 ts_tablet_manager.cc:1918] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:09.264966 19107 log.cc:1196] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/3937f3e1b5ab43a9a40d312174a001a8
I20260806 23:52:09.265349 19107 ts_tablet_manager.cc:1939] T 3937f3e1b5ab43a9a40d312174a001a8 P 13d183295820422888516cabcc71ddc2: Deleting consensus metadata
I20260806 23:52:09.265470 18338 catalog_manager.cc:4841] TS b550b0a6d3474992905775e5f730c524 (127.10.225.131:35309): tablet 3937f3e1b5ab43a9a40d312174a001a8 (table test-table [id=5fcd74f5be4e40dc87ebda0cdcd1d4e3]) successfully deleted
I20260806 23:52:09.266460 18333 catalog_manager.cc:4841] TS 13d183295820422888516cabcc71ddc2 (127.10.225.130:41019): tablet 3937f3e1b5ab43a9a40d312174a001a8 (table test-table [id=5fcd74f5be4e40dc87ebda0cdcd1d4e3]) successfully deleted
W20260806 23:52:09.266713 18333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:41797 (error 108)
I20260806 23:52:09.267001 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 18649
I20260806 23:52:09.273033 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.267489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:41797 (local address 127.0.0.1:54152)
0806 23:52:09.272827 (+ 5338us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:09.272929 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:41797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5211}
W20260806 23:52:09.273994 18333 catalog_manager.cc:4584] TS 79958e0ed478424a98f35a1c1fa997d9 (127.10.225.129:41797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3937f3e1b5ab43a9a40d312174a001a8: Network error: Client connection negotiation failed: client connection to 127.10.225.129:41797: connect: Connection refused (error 111)
I20260806 23:52:09.280092 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 18848
I20260806 23:52:09.295543 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 18308
2026-08-06T23:52:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1168 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 23:52:09.325169 11142 mini_oidc.cc:152] Starting JWKS server
I20260806 23:52:09.327193 11142 webserver.cc:460] Webserver started at https://127.0.0.1:41437/ using document root <none> and password file <none>
I20260806 23:52:09.327363 11142 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 23:52:09.327867 11142 webserver.cc:460] Webserver started at http://127.0.0.1:42787/ using document root <none> and password file <none>
2026-08-06T23:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:09Z Disabled control of system clock
I20260806 23:52:09.339496 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38443
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:34831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:38443
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41437/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/testchainca.cert with env {}
W20260806 23:52:09.465178 19120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:09.465468 19120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:09.465509 19120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:09.469987 19120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:09.470121 19120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:09.470139 19120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:09.470158 19120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:09.470175 19120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 23:52:09.470196 19120 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 23:52:09.470225 19120 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41437/jwks.json
I20260806 23:52:09.478112 19120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:38443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41437/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:09.479377 19120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:09.480476 19120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:09.489388 19127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:09.489400 19130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:09.489400 19128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:09.489748 19120 server_base.cc:1034] running on GCE node
I20260806 23:52:09.490746 19120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:09.492084 19120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:09.493280 19120 hybrid_clock.cc:648] HybridClock initialized: now 1786060329493259 us; error 41 us; skew 500 ppm
I20260806 23:52:09.495374 19120 webserver.cc:460] Webserver started at http://127.10.225.190:41977/ using document root <none> and password file <none>
I20260806 23:52:09.496032 19120 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:09.496105 19120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:09.496313 19120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:09.498415 19120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "69abf9f1edf44c03b65fa2451df017a5"
format_stamp: "Formatted at 2026-08-06 23:52:09 on dist-test-slave-nj21"
server_key: "e51ec094c9753799ed6678f610763d59"
server_key_iv: "5663f8ed3b9e0d801d8abe05716edb66"
server_key_version: "encryptionkey@0"
I20260806 23:52:09.498987 19120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "69abf9f1edf44c03b65fa2451df017a5"
format_stamp: "Formatted at 2026-08-06 23:52:09 on dist-test-slave-nj21"
server_key: "e51ec094c9753799ed6678f610763d59"
server_key_iv: "5663f8ed3b9e0d801d8abe05716edb66"
server_key_version: "encryptionkey@0"
I20260806 23:52:09.503363 19120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260806 23:52:09.506726 19141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:09.508183 19120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:09.508324 19120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "69abf9f1edf44c03b65fa2451df017a5"
format_stamp: "Formatted at 2026-08-06 23:52:09 on dist-test-slave-nj21"
server_key: "e51ec094c9753799ed6678f610763d59"
server_key_iv: "5663f8ed3b9e0d801d8abe05716edb66"
server_key_version: "encryptionkey@0"
I20260806 23:52:09.508451 19120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:09.528837 19120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:09.529632 19120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:09.529817 19120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:09.536972 19120 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:38443
I20260806 23:52:09.536990 19221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:38443 every 8 connection(s)
I20260806 23:52:09.538000 19120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:09.538923 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 19120
I20260806 23:52:09.539033 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:09.539343 11142 external_mini_cluster.cc:1428] Setting key cf34eabee35f1db3c74c52dc3a5c1773
I20260806 23:52:09.542018 19222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:09.547863 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5: Bootstrap starting.
I20260806 23:52:09.549044 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.541064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57476 (local address 127.10.225.190:38443)
0806 23:52:09.541870 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.541890 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.541939 (+ 49us) server_negotiation.cc:407] Connection header received
0806 23:52:09.542645 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.542662 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.543068 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.543394 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.543856 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:09.544812 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:09.545711 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:09.546184 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:09.546332 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:09.547227 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.547232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.547246 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.547620 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.547703 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.547731 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.547825 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.547954 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.548308 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":157,"threads_started":1}
I20260806 23:52:09.550308 19222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:09.551168 19222 log.cc:824] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:09.552011 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:52:09.553206 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5: No bootstrap required, opened a new log
I20260806 23:52:09.556984 19222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69abf9f1edf44c03b65fa2451df017a5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38443 } }
I20260806 23:52:09.557211 19222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:09.557246 19222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69abf9f1edf44c03b65fa2451df017a5, State: Initialized, Role: FOLLOWER
I20260806 23:52:09.557631 19222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69abf9f1edf44c03b65fa2451df017a5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38443 } }
I20260806 23:52:09.557757 19222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:09.557798 19222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:09.557864 19222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:09.558776 19222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69abf9f1edf44c03b65fa2451df017a5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38443 } }
I20260806 23:52:09.559101 19222 leader_election.cc:302] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69abf9f1edf44c03b65fa2451df017a5; no voters:
I20260806 23:52:09.559378 19222 leader_election.cc:288] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:09.559518 19232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:09.559743 19232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [term 1 LEADER]: Becoming Leader. State: Replica: 69abf9f1edf44c03b65fa2451df017a5, State: Running, Role: LEADER
I20260806 23:52:09.560076 19232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69abf9f1edf44c03b65fa2451df017a5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38443 } }
I20260806 23:52:09.560765 19222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:09.561746 19233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69abf9f1edf44c03b65fa2451df017a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69abf9f1edf44c03b65fa2451df017a5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38443 } } }
I20260806 23:52:09.561759 19234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69abf9f1edf44c03b65fa2451df017a5. Latest consensus state: current_term: 1 leader_uuid: "69abf9f1edf44c03b65fa2451df017a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69abf9f1edf44c03b65fa2451df017a5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38443 } } }
I20260806 23:52:09.561893 19233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:09.561893 19234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:09.563097 19247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:09.566231 19247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:09.572453 19247 catalog_manager.cc:1347] Generated new cluster ID: cb02a026630f4b45a645f1bf05497fbc
I20260806 23:52:09.572531 19247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:09.589675 19247 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:09.591044 19247 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:09.602329 19247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69abf9f1edf44c03b65fa2451df017a5: Generated new TSK 0
I20260806 23:52:09.603125 19247 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 23:52:09.641400 19273 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 23:52:09.641655 19273 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 23:52:09.642192 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.622673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.10.225.190:38443)
0806 23:52:09.622820 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.622825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.622954 (+ 129us) server_negotiation.cc:407] Connection header received
0806 23:52:09.623172 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.623176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.623241 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.623362 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0806 23:52:09.623818 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:09.624869 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:09.625928 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:09.626289 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:09.626393 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:09.626748 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 23:52:09.641713 (+ 14965us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 23:52:09.642038 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 23:52:09.642266 19226 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 23:52:09.643893 19270 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:09.622534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38443 (local address 127.0.0.1:57488)
0806 23:52:09.622870 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:09.622883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:09.623002 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:09.623417 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:09.623420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:09.623429 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 23:52:09.623558 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:09.623564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:09.625019 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:09.625022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:09.625806 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:09.625816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:09.626412 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:09.626415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:09.626485 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:09.626540 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 23:52:09.641976 (+ 15436us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 23:52:09.642332 (+ 356us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.225.190:38443: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.225.190:38443: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 23:52:09.644817 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 19120
2026-08-06T23:52:09Z chronyd exiting
I20260806 23:52:09.693421 11142 test_util.cc:175] -----------------------------------------------
I20260806 23:52:09.693532 11142 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786060299689793-11142-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (370 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 23:52:09.694757 11142 mini_oidc.cc:152] Starting JWKS server
I20260806 23:52:09.695628 11142 webserver.cc:460] Webserver started at https://127.0.0.1:44501/ using document root <none> and password file <none>
I20260806 23:52:09.695760 11142 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 23:52:09.695993 11142 webserver.cc:460] Webserver started at http://127.0.0.1:37799/ using document root <none> and password file <none>
2026-08-06T23:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:09Z Disabled control of system clock
I20260806 23:52:09.707237 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:42341
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:42341
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/testchainca.cert with env {}
W20260806 23:52:09.831636 19281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:09.831991 19281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:09.832058 19281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:09.836440 19281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:09.836566 19281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:09.836596 19281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:09.836621 19281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:09.836653 19281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 23:52:09.836680 19281 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 23:52:09.836707 19281 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44501/jwks.json
I20260806 23:52:09.846300 19281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:42341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:09.847795 19281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:09.848841 19281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:09.857791 19292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:09.857921 19291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:09.857791 19294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:09.858060 19281 server_base.cc:1034] running on GCE node
I20260806 23:52:09.858673 19281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:09.859954 19281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:09.861138 19281 hybrid_clock.cc:648] HybridClock initialized: now 1786060329861115 us; error 33 us; skew 500 ppm
I20260806 23:52:09.863152 19281 webserver.cc:460] Webserver started at http://127.10.225.190:34123/ using document root <none> and password file <none>
I20260806 23:52:09.863803 19281 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:09.863881 19281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:09.864115 19281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:09.866271 19281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "28b594d6e57840b5a2dc024867363dac"
format_stamp: "Formatted at 2026-08-06 23:52:09 on dist-test-slave-nj21"
server_key: "0e9897c15edadaa82789475138bafaf7"
server_key_iv: "4eeed7c30ae4804cae11e0529194e0f9"
server_key_version: "encryptionkey@0"
I20260806 23:52:09.866827 19281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "28b594d6e57840b5a2dc024867363dac"
format_stamp: "Formatted at 2026-08-06 23:52:09 on dist-test-slave-nj21"
server_key: "0e9897c15edadaa82789475138bafaf7"
server_key_iv: "4eeed7c30ae4804cae11e0529194e0f9"
server_key_version: "encryptionkey@0"
I20260806 23:52:09.870872 19281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:09.874185 19304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:09.875851 19281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:09.875994 19281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "28b594d6e57840b5a2dc024867363dac"
format_stamp: "Formatted at 2026-08-06 23:52:09 on dist-test-slave-nj21"
server_key: "0e9897c15edadaa82789475138bafaf7"
server_key_iv: "4eeed7c30ae4804cae11e0529194e0f9"
server_key_version: "encryptionkey@0"
I20260806 23:52:09.876114 19281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:09.912480 19281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:09.913249 19281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:09.913442 19281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:09.922045 19281 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:42341
I20260806 23:52:09.922158 19374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:42341 every 8 connection(s)
I20260806 23:52:09.923501 19281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:09.926234 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 19281
I20260806 23:52:09.926388 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:09.926673 11142 external_mini_cluster.cc:1428] Setting key 24b2bdeb74f0f0820da36d7b1290d0dd
I20260806 23:52:09.927389 19375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:09.933984 19375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac: Bootstrap starting.
I20260806 23:52:09.936039 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.928200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.10.225.190:42341)
0806 23:52:09.928923 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.928946 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.928992 (+ 46us) server_negotiation.cc:407] Connection header received
0806 23:52:09.929707 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.929723 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.930124 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.930428 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:09.930882 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:09.931931 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:09.932800 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:09.933278 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:09.933426 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:09.934256 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:09.934262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:09.934275 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:09.934674 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:09.934730 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:09.934751 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:09.934838 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:09.934980 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 23:52:09.935243 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
I20260806 23:52:09.936434 19375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:09.937366 19375 log.cc:824] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:09.939726 19375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac: No bootstrap required, opened a new log
I20260806 23:52:09.939997 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:52:09.943702 19375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b594d6e57840b5a2dc024867363dac" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42341 } }
I20260806 23:52:09.943953 19375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:09.944024 19375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b594d6e57840b5a2dc024867363dac, State: Initialized, Role: FOLLOWER
I20260806 23:52:09.944460 19375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b594d6e57840b5a2dc024867363dac" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42341 } }
I20260806 23:52:09.944597 19375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:09.944664 19375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:09.944763 19375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:09.945698 19375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b594d6e57840b5a2dc024867363dac" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42341 } }
I20260806 23:52:09.946029 19375 leader_election.cc:302] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28b594d6e57840b5a2dc024867363dac; no voters:
I20260806 23:52:09.946332 19375 leader_election.cc:288] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:09.946556 19386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:09.946810 19386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [term 1 LEADER]: Becoming Leader. State: Replica: 28b594d6e57840b5a2dc024867363dac, State: Running, Role: LEADER
I20260806 23:52:09.947154 19386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b594d6e57840b5a2dc024867363dac" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42341 } }
I20260806 23:52:09.947754 19375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:09.948814 19388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28b594d6e57840b5a2dc024867363dac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b594d6e57840b5a2dc024867363dac" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42341 } } }
I20260806 23:52:09.948972 19388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:09.948814 19389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b594d6e57840b5a2dc024867363dac. Latest consensus state: current_term: 1 leader_uuid: "28b594d6e57840b5a2dc024867363dac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b594d6e57840b5a2dc024867363dac" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42341 } } }
I20260806 23:52:09.949201 19389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:09.949506 19401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:09.953372 19401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:09.959726 19401 catalog_manager.cc:1347] Generated new cluster ID: 3cd87a4efd254ea6a8edf922279752a6
I20260806 23:52:09.959851 19401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:09.970026 19401 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:09.971813 19401 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:09.978891 19401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28b594d6e57840b5a2dc024867363dac: Generated new TSK 0
I20260806 23:52:09.980371 19401 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 23:52:10.021567 19424 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 23:52:10.021672 19424 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 23:52:10.022457 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:09.998478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.10.225.190:42341)
0806 23:52:09.998657 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:09.998663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:09.998941 (+ 278us) server_negotiation.cc:407] Connection header received
0806 23:52:09.999172 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:09.999176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:09.999238 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:09.999461 (+ 223us) server_negotiation.cc:226] Negotiated authn=JWT
0806 23:52:09.999808 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.000567 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.001985 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.002260 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.002388 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:10.002826 (+ 438us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 23:52:10.021981 (+ 19155us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 23:52:10.022288 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 23:52:10.022531 19383 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 23:52:10.022697 19423 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:09.998343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:42341 (local address 127.0.0.1:56096)
0806 23:52:09.998785 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:09.998803 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:09.998961 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:09.999449 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:09.999452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:09.999460 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 23:52:09.999615 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:09.999621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.000701 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.000705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:10.001807 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:10.001819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.002421 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.002425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:10.002523 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:10.002559 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 23:52:10.022225 (+ 19666us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 23:52:10.022491 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.225.190:42341: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":128,"threads_started":1}
I20260806 23:52:10.023465 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 19281
2026-08-06T23:52:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (380 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 23:52:10.075810 11142 mini_oidc.cc:152] Starting JWKS server
I20260806 23:52:10.076727 11142 webserver.cc:460] Webserver started at https://127.0.0.1:42835/ using document root <none> and password file <none>
I20260806 23:52:10.076814 11142 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 23:52:10.077085 11142 webserver.cc:460] Webserver started at http://127.0.0.1:34983/ using document root <none> and password file <none>
2026-08-06T23:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:10Z Disabled control of system clock
I20260806 23:52:10.089447 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38341
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:42583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:38341
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42835/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 23:52:10.208825 19432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:10.209112 19432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:10.209153 19432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:10.213511 19432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:10.213595 19432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:10.213610 19432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:10.213627 19432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:10.213642 19432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 23:52:10.213658 19432 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 23:52:10.213678 19432 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42835/jwks.json
I20260806 23:52:10.218731 19432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:38341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42835/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:10.219939 19432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:10.221395 19432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:10.230140 19444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:10.230139 19446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:10.230352 19432 server_base.cc:1034] running on GCE node
W20260806 23:52:10.230151 19448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:10.231036 19432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:10.232231 19432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:10.233497 19432 hybrid_clock.cc:648] HybridClock initialized: now 1786060330233475 us; error 55 us; skew 500 ppm
I20260806 23:52:10.235581 19432 webserver.cc:460] Webserver started at http://127.10.225.190:46747/ using document root <none> and password file <none>
I20260806 23:52:10.236214 19432 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:10.236287 19432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:10.236498 19432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:10.238647 19432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "36938d7769424b6292bc9735dbd68876"
format_stamp: "Formatted at 2026-08-06 23:52:10 on dist-test-slave-nj21"
server_key: "8d8a7c979ef8e2f2b4b87920e01e9635"
server_key_iv: "08af9a8693d6258c95b04c7602b9e50d"
server_key_version: "encryptionkey@0"
I20260806 23:52:10.239202 19432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "36938d7769424b6292bc9735dbd68876"
format_stamp: "Formatted at 2026-08-06 23:52:10 on dist-test-slave-nj21"
server_key: "8d8a7c979ef8e2f2b4b87920e01e9635"
server_key_iv: "08af9a8693d6258c95b04c7602b9e50d"
server_key_version: "encryptionkey@0"
I20260806 23:52:10.243042 19432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:10.246654 19456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:10.248133 19432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:10.248270 19432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "36938d7769424b6292bc9735dbd68876"
format_stamp: "Formatted at 2026-08-06 23:52:10 on dist-test-slave-nj21"
server_key: "8d8a7c979ef8e2f2b4b87920e01e9635"
server_key_iv: "08af9a8693d6258c95b04c7602b9e50d"
server_key_version: "encryptionkey@0"
I20260806 23:52:10.248396 19432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:10.284184 19432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:10.285148 19432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:10.285363 19432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:10.294667 19432 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:38341
I20260806 23:52:10.294759 19529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:38341 every 8 connection(s)
I20260806 23:52:10.295800 19432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:10.298230 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 19432
I20260806 23:52:10.298360 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:10.298646 11142 external_mini_cluster.cc:1428] Setting key a7a056bdb4d2c8d89e92530aca34bc1f
I20260806 23:52:10.299998 19530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:10.309128 19530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876: Bootstrap starting.
I20260806 23:52:10.311064 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:10.300218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.10.225.190:38341)
0806 23:52:10.301161 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:10.301188 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:10.301247 (+ 59us) server_negotiation.cc:407] Connection header received
0806 23:52:10.302236 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:10.302263 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:10.302880 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:10.303361 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:10.303942 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.305490 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.306776 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.307661 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.307912 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:10.308783 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:10.308790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:10.308816 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:10.309349 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:10.309434 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:10.309478 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:10.309577 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:10.309747 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 23:52:10.310055 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":142,"threads_started":1}
I20260806 23:52:10.312137 19530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:10.312896 19530 log.cc:824] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:10.314810 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:52:10.315466 19530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876: No bootstrap required, opened a new log
I20260806 23:52:10.319408 19530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36938d7769424b6292bc9735dbd68876" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38341 } }
I20260806 23:52:10.319741 19530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:10.319815 19530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36938d7769424b6292bc9735dbd68876, State: Initialized, Role: FOLLOWER
I20260806 23:52:10.320278 19530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36938d7769424b6292bc9735dbd68876" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38341 } }
I20260806 23:52:10.320423 19530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:10.320492 19530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:10.320590 19530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:10.321619 19530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36938d7769424b6292bc9735dbd68876" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38341 } }
I20260806 23:52:10.321970 19530 leader_election.cc:302] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36938d7769424b6292bc9735dbd68876; no voters:
I20260806 23:52:10.322324 19530 leader_election.cc:288] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:10.322444 19538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:10.322681 19538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [term 1 LEADER]: Becoming Leader. State: Replica: 36938d7769424b6292bc9735dbd68876, State: Running, Role: LEADER
I20260806 23:52:10.323004 19538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36938d7769424b6292bc9735dbd68876" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38341 } }
I20260806 23:52:10.323632 19530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:10.324710 19539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36938d7769424b6292bc9735dbd68876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36938d7769424b6292bc9735dbd68876" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38341 } } }
I20260806 23:52:10.324865 19539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:10.325119 19541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36938d7769424b6292bc9735dbd68876. Latest consensus state: current_term: 1 leader_uuid: "36938d7769424b6292bc9735dbd68876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36938d7769424b6292bc9735dbd68876" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38341 } } }
I20260806 23:52:10.325222 19541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:10.325349 19550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:10.328558 19550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:10.334511 19550 catalog_manager.cc:1347] Generated new cluster ID: 8abc21ec6b89478d9d271627f56d2fe4
I20260806 23:52:10.334595 19550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:10.355535 19550 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:10.356931 19550 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:10.375948 19550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36938d7769424b6292bc9735dbd68876: Generated new TSK 0
I20260806 23:52:10.376869 19550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:52:10.396565 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:10.390277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.10.225.190:38341)
0806 23:52:10.390532 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:10.390537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:10.390582 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:52:10.390744 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:10.390748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:10.390835 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:10.390978 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0806 23:52:10.391423 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.392432 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.393455 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.393982 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.394104 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:10.394477 (+ 373us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 23:52:10.396154 (+ 1677us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 23:52:10.396404 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34770: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 23:52:10.396648 19534 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34770: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 23:52:10.398114 19571 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:10.389317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:38341 (local address 127.0.0.1:34770)
0806 23:52:10.389615 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:10.389629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:10.389734 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:10.391125 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:10.391130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:10.391140 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0806 23:52:10.391271 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:10.391298 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.392599 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.392602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:10.393260 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:10.393269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.394141 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.394144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:10.394214 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:10.394242 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 23:52:10.396330 (+ 2088us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 23:52:10.397927 (+ 1597us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.225.190:38341: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":1402,"thread_start_us":92,"threads_started":1}
I20260806 23:52:10.398756 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 19432
E20260806 23:52:10.420368 19572 webserver.cc:533] Webserver: sslize failed:
E20260806 23:52:10.420493 19572 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T23:52:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (377 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T23:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:10Z Disabled control of system clock
I20260806 23:52:10.464030 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:43781
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:43781
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/testchainca.cert with env {}
W20260806 23:52:10.583658 19579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:10.583974 19579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:10.584018 19579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:10.588109 19579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:10.588176 19579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:10.588189 19579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:10.588204 19579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:10.588218 19579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 23:52:10.588233 19579 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 23:52:10.588245 19579 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 23:52:10.593930 19579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:43781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:10.595062 19579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:10.596177 19579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:10.605010 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:10.605029 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:10.605023 19589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:10.605362 19579 server_base.cc:1034] running on GCE node
I20260806 23:52:10.606135 19579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:10.607357 19579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:10.608549 19579 hybrid_clock.cc:648] HybridClock initialized: now 1786060330608543 us; error 69 us; skew 500 ppm
I20260806 23:52:10.610883 19579 webserver.cc:460] Webserver started at http://127.10.225.190:42001/ using document root <none> and password file <none>
I20260806 23:52:10.611706 19579 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:10.611809 19579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:10.612088 19579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:10.615118 19579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "cbe14d31933d4d79b3b957376bda776d"
format_stamp: "Formatted at 2026-08-06 23:52:10 on dist-test-slave-nj21"
server_key: "8f94ffed28c7ae87501d289344b4648e"
server_key_iv: "6f8b7d3bff6fb632ab422faeb50d00dc"
server_key_version: "encryptionkey@0"
I20260806 23:52:10.615918 19579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "cbe14d31933d4d79b3b957376bda776d"
format_stamp: "Formatted at 2026-08-06 23:52:10 on dist-test-slave-nj21"
server_key: "8f94ffed28c7ae87501d289344b4648e"
server_key_iv: "6f8b7d3bff6fb632ab422faeb50d00dc"
server_key_version: "encryptionkey@0"
I20260806 23:52:10.622080 19579 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 23:52:10.625279 19602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:10.626684 19579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:52:10.626808 19579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "cbe14d31933d4d79b3b957376bda776d"
format_stamp: "Formatted at 2026-08-06 23:52:10 on dist-test-slave-nj21"
server_key: "8f94ffed28c7ae87501d289344b4648e"
server_key_iv: "6f8b7d3bff6fb632ab422faeb50d00dc"
server_key_version: "encryptionkey@0"
I20260806 23:52:10.626958 19579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:10.655406 19579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:10.656291 19579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:10.656482 19579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:10.663763 19579 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:43781
I20260806 23:52:10.663802 19689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:43781 every 8 connection(s)
I20260806 23:52:10.664739 19579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:10.668788 19690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:10.673100 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 19579
I20260806 23:52:10.673267 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:10.673637 11142 external_mini_cluster.cc:1428] Setting key a5bed5c702ed84ad7a3702b96e9e4ea4
I20260806 23:52:10.676709 19690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d: Bootstrap starting.
I20260806 23:52:10.679313 19690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:10.680156 19690 log.cc:824] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:10.682257 19690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d: No bootstrap required, opened a new log
I20260806 23:52:10.684695 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:10.675545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.10.225.190:43781)
0806 23:52:10.676649 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:10.676672 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:10.676721 (+ 49us) server_negotiation.cc:407] Connection header received
0806 23:52:10.677574 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:10.677594 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:10.678045 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:10.678400 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:10.678898 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.680084 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.681046 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.681542 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.681706 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:10.682632 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:10.682637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:10.682652 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:10.683065 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:10.683126 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:10.683148 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:10.683260 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:10.683399 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 23:52:10.683729 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":192,"threads_started":1}
I20260806 23:52:10.686625 19690 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe14d31933d4d79b3b957376bda776d" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43781 } }
I20260806 23:52:10.686928 19690 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:10.686991 19690 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe14d31933d4d79b3b957376bda776d, State: Initialized, Role: FOLLOWER
I20260806 23:52:10.687445 19690 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe14d31933d4d79b3b957376bda776d" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43781 } }
I20260806 23:52:10.687582 19690 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:10.687637 19690 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:10.687773 19690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:10.687883 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:52:10.688804 19690 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe14d31933d4d79b3b957376bda776d" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43781 } }
I20260806 23:52:10.689175 19690 leader_election.cc:302] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbe14d31933d4d79b3b957376bda776d; no voters:
I20260806 23:52:10.689585 19690 leader_election.cc:288] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:10.689674 19696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:10.689854 19696 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [term 1 LEADER]: Becoming Leader. State: Replica: cbe14d31933d4d79b3b957376bda776d, State: Running, Role: LEADER
I20260806 23:52:10.690153 19696 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe14d31933d4d79b3b957376bda776d" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43781 } }
I20260806 23:52:10.690953 19690 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:10.692219 19697 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbe14d31933d4d79b3b957376bda776d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe14d31933d4d79b3b957376bda776d" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43781 } } }
I20260806 23:52:10.692265 19699 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbe14d31933d4d79b3b957376bda776d. Latest consensus state: current_term: 1 leader_uuid: "cbe14d31933d4d79b3b957376bda776d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe14d31933d4d79b3b957376bda776d" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43781 } } }
I20260806 23:52:10.692363 19697 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:10.692370 19699 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:10.692853 19711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:10.695721 19711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:10.702266 19711 catalog_manager.cc:1347] Generated new cluster ID: cfdde45e6c5545d2b8b3b6ba59dc26cc
I20260806 23:52:10.702368 19711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:10.729540 19711 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:10.730973 19711 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:10.741423 19711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbe14d31933d4d79b3b957376bda776d: Generated new TSK 0
I20260806 23:52:10.742393 19711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:52:11.654373 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:10.760202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.10.225.190:43781)
0806 23:52:10.760455 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:10.760460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:10.760476 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:52:10.760612 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:10.760615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:10.760667 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:10.760764 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0806 23:52:10.761006 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.761668 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.762554 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.762777 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.762864 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:10.763201 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 23:52:11.653847 (+890646us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 23:52:11.654206 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55160: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":131}
W20260806 23:52:11.654490 19694 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55160: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 23:52:11.654569 19736 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:10.759985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:43781 (local address 127.0.0.1:55160)
0806 23:52:10.760260 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:10.760357 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:10.760485 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:10.760772 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:10.760775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:10.760783 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 23:52:10.760896 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:10.760903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.761807 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.761811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:10.762440 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:10.762448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:10.762896 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:10.762900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:10.762997 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:10.763023 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 23:52:11.654156 (+891133us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 23:52:11.654370 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.225.190:43781: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
I20260806 23:52:11.655237 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 19579
2026-08-06T23:52:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1230 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (150 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (147 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:52:12 dist-test-slave-nj21 krb5kdc[19759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:52:12 dist-test-slave-nj21 krb5kdc[19759](info): set up 2 sockets
Aug 06 23:52:12 dist-test-slave-nj21 krb5kdc[19759](info): commencing operation
krb5kdc: starting...
W20260806 23:52:13.123401 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:13 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:52:13.261421 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:46589
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:46589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:13.386678 19778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:13.386986 19778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:13.387046 19778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:13.391218 19778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:13.391295 19778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:13.391324 19778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:13.391346 19778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:13.391367 19778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:13.398056 19778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:46589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:13.399870 19778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:13.401298 19778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:13.411770 19788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:13.411875 19778 server_base.cc:1034] running on GCE node
W20260806 23:52:13.411886 19784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:13.411772 19785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:13.412653 19778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:13.413838 19778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:13.415099 19778 hybrid_clock.cc:648] HybridClock initialized: now 1786060333415059 us; error 34 us; skew 500 ppm
Aug 06 23:52:13 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060333, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:13.418762 19778 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:52:13.419775 19778 webserver.cc:460] Webserver started at http://127.10.225.190:46091/ using document root <none> and password file <none>
I20260806 23:52:13.420356 19778 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:13.420418 19778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:13.420634 19778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:13.422708 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "8be5a5b417a1490985aac6affdb7f27f"
format_stamp: "Formatted at 2026-08-06 23:52:13 on dist-test-slave-nj21"
server_key: "507179ae0df09b28748edeeb44dc64b4"
server_key_iv: "d17f4d1cf88cd02284f2d42559738d0b"
server_key_version: "encryptionkey@0"
I20260806 23:52:13.423244 19778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "8be5a5b417a1490985aac6affdb7f27f"
format_stamp: "Formatted at 2026-08-06 23:52:13 on dist-test-slave-nj21"
server_key: "507179ae0df09b28748edeeb44dc64b4"
server_key_iv: "d17f4d1cf88cd02284f2d42559738d0b"
server_key_version: "encryptionkey@0"
I20260806 23:52:13.427515 19778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:13.430047 19803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:13.431282 19778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:52:13.431406 19778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "8be5a5b417a1490985aac6affdb7f27f"
format_stamp: "Formatted at 2026-08-06 23:52:13 on dist-test-slave-nj21"
server_key: "507179ae0df09b28748edeeb44dc64b4"
server_key_iv: "d17f4d1cf88cd02284f2d42559738d0b"
server_key_version: "encryptionkey@0"
I20260806 23:52:13.431524 19778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:13.445868 19778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:13.450006 19778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:13.450219 19778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:13.457554 19778 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:46589
I20260806 23:52:13.457572 19879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:46589 every 8 connection(s)
I20260806 23:52:13.458552 19778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:13.461378 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 19778
I20260806 23:52:13.461542 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:13.461920 11142 external_mini_cluster.cc:1428] Setting key 7a5b538427dab1025ea4f4c16ef64e9e
I20260806 23:52:13.462810 19880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:13.469856 19880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f: Bootstrap starting.
Aug 06 23:52:13 dist-test-slave-nj21 krb5kdc[19759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:13.473086 19880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:13.474046 19880 log.cc:824] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:13.477058 19880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f: No bootstrap required, opened a new log
I20260806 23:52:13.479230 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:13.463848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.10.225.190:46589)
0806 23:52:13.464492 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:13.464512 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:13.464559 (+ 47us) server_negotiation.cc:407] Connection header received
0806 23:52:13.465291 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:13.465311 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:13.465699 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:13.466048 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:13.466910 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:13.468444 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:13.469519 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:13.470011 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:13.470166 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:13.472988 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:13.472994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:13.473009 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:13.473040 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:13.475257 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:13.475855 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:13.475862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:13.475869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:13.476109 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:13.476457 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:13.476461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:13.476463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:13.477111 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:13.477315 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:13.477619 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 23:52:13.477970 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":203,"threads_started":1}
I20260806 23:52:13.481153 19880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be5a5b417a1490985aac6affdb7f27f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46589 } }
I20260806 23:52:13.481400 19880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:13.481449 19880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be5a5b417a1490985aac6affdb7f27f, State: Initialized, Role: FOLLOWER
I20260806 23:52:13.481894 19880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be5a5b417a1490985aac6affdb7f27f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46589 } }
I20260806 23:52:13.482017 19880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:13.482062 19880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:13.482132 19880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:13.483071 19880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be5a5b417a1490985aac6affdb7f27f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46589 } }
I20260806 23:52:13.483383 19880 leader_election.cc:302] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8be5a5b417a1490985aac6affdb7f27f; no voters:
I20260806 23:52:13.483659 19880 leader_election.cc:288] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:13.483913 19886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:13.484258 19886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [term 1 LEADER]: Becoming Leader. State: Replica: 8be5a5b417a1490985aac6affdb7f27f, State: Running, Role: LEADER
I20260806 23:52:13.484668 19886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be5a5b417a1490985aac6affdb7f27f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46589 } }
I20260806 23:52:13.485028 19880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:13.486683 19890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8be5a5b417a1490985aac6affdb7f27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be5a5b417a1490985aac6affdb7f27f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46589 } } }
I20260806 23:52:13.486850 19890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:13.487432 19891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8be5a5b417a1490985aac6affdb7f27f. Latest consensus state: current_term: 1 leader_uuid: "8be5a5b417a1490985aac6affdb7f27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be5a5b417a1490985aac6affdb7f27f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46589 } } }
I20260806 23:52:13.487537 19891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f [sys.catalog]: This master's current role is: LEADER
W20260806 23:52:13.490756 19906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:52:13.490837 19906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:52:13.490922 19901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:13.491814 19901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:13.498425 19901 catalog_manager.cc:1347] Generated new cluster ID: 1000d549681143adbc6b7916fd6a2c84
I20260806 23:52:13.498737 19901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:13.520597 19901 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:13.521654 19901 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:13.531584 19901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8be5a5b417a1490985aac6affdb7f27f: Generated new TSK 0
I20260806 23:52:13.532503 19901 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:52:13.567651 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46589
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:13.696635 19916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:13.696967 19916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:13.697017 19916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:13.701253 19916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:13.701344 19916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:13.701408 19916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:13.707988 19916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:13.709234 19916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:13.710413 19916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:13.721060 19928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:13.721092 19930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:13.721163 19916 server_base.cc:1034] running on GCE node
W20260806 23:52:13.721081 19927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:13.721947 19916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:13.722820 19916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:13.724128 19916 hybrid_clock.cc:648] HybridClock initialized: now 1786060333724066 us; error 55 us; skew 500 ppm
Aug 06 23:52:13 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060333, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:13.727353 19916 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:52:13.728474 19916 webserver.cc:460] Webserver started at http://127.10.225.129:43325/ using document root <none> and password file <none>
I20260806 23:52:13.729103 19916 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:13.729172 19916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:13.729379 19916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:13.731567 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "77d01b62896148f5a0a750e05ebb96f6"
format_stamp: "Formatted at 2026-08-06 23:52:13 on dist-test-slave-nj21"
server_key: "7cd6638705c98c3388094797a46dca6d"
server_key_iv: "6072c0392dfa229f336f33dcd1705410"
server_key_version: "encryptionkey@0"
I20260806 23:52:13.732177 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "77d01b62896148f5a0a750e05ebb96f6"
format_stamp: "Formatted at 2026-08-06 23:52:13 on dist-test-slave-nj21"
server_key: "7cd6638705c98c3388094797a46dca6d"
server_key_iv: "6072c0392dfa229f336f33dcd1705410"
server_key_version: "encryptionkey@0"
I20260806 23:52:13.736212 19916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:52:13.739146 19944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:13.740505 19916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:13.740625 19916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "77d01b62896148f5a0a750e05ebb96f6"
format_stamp: "Formatted at 2026-08-06 23:52:13 on dist-test-slave-nj21"
server_key: "7cd6638705c98c3388094797a46dca6d"
server_key_iv: "6072c0392dfa229f336f33dcd1705410"
server_key_version: "encryptionkey@0"
I20260806 23:52:13.740732 19916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:13.754977 19916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:13.759003 19916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:13.759218 19916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:13.759929 19916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:13.760864 19916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:13.760915 19916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:13.760973 19916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:13.761003 19916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:13.770089 19916 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:45041
I20260806 23:52:13.770128 20114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:45041 every 8 connection(s)
I20260806 23:52:13.771052 19916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:13.776222 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 19916
I20260806 23:52:13.776341 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:13.776613 11142 external_mini_cluster.cc:1428] Setting key 56fc49ad2fe3a619a2236dbd8e47e047
Aug 06 23:52:13 dist-test-slave-nj21 krb5kdc[19759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060333, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:13.787168 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:13.773099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:36513 (local address 127.10.225.190:46589)
0806 23:52:13.773245 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:13.773248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:13.774154 (+ 906us) server_negotiation.cc:407] Connection header received
0806 23:52:13.775594 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:13.775597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:13.775700 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:13.775796 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:13.777206 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:13.777855 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:13.779099 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:13.779342 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:13.779444 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:13.782549 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:13.782555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:13.782559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:13.782611 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:13.784575 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:13.785174 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:13.785178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:13.785181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:13.785292 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:13.785778 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:13.785783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:13.785784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:13.786094 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:13.786220 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:13.786796 (+ 576us) server_negotiation.cc:299] Negotiation successful
0806 23:52:13.786965 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:52:13.788062 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:13.773451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46589 (local address 127.10.225.129:36513)
0806 23:52:13.773966 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:13.774012 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:13.775330 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:13.775984 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:13.775992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:13.776464 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:13.777036 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:13.777053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:13.777988 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:13.777992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:13.778962 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:13.778974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:13.779504 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:13.779509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:13.779743 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:13.780372 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:13.780391 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:13.782200 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:13.784738 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:13.784745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:13.784765 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:13.785062 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:13.785439 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:13.785442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:13.785443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:13.785664 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:13.786239 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:13.786246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:13.786530 (+ 284us) client_negotiation.cc:769] Sending connection context
0806 23:52:13.786708 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 23:52:13.787058 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:52:13.789389 20115 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46589
I20260806 23:52:13.789649 20115 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:13.790182 20115 heartbeater.cc:511] Master 127.10.225.190:46589 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:52:13.791978 19823 ts_manager.cc:194] Registered new tserver with Master: 77d01b62896148f5a0a750e05ebb96f6 (127.10.225.129:45041)
I20260806 23:52:13.793200 19823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:36513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:52:13.810544 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46589
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:13.944687 20127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:13.945020 20127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:13.945083 20127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:13.949276 20127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:13.949357 20127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:13.949424 20127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:13.955032 20127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:13.956287 20127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:13.957396 20127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:13.967626 20139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:13.967626 20136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:13.967783 20137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:13.968187 20127 server_base.cc:1034] running on GCE node
I20260806 23:52:13.968670 20127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:13.969426 20127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:13.970644 20127 hybrid_clock.cc:648] HybridClock initialized: now 1786060333970610 us; error 43 us; skew 500 ppm
Aug 06 23:52:13 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060333, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:13.973721 20127 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:52:13.974781 20127 webserver.cc:460] Webserver started at http://127.10.225.130:35357/ using document root <none> and password file <none>
I20260806 23:52:13.975404 20127 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:13.975483 20127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:13.975780 20127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:13.978765 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "734107795ec84e428301526718949b13"
format_stamp: "Formatted at 2026-08-06 23:52:13 on dist-test-slave-nj21"
server_key: "1d94f990e6d66e1619755eee678e9a17"
server_key_iv: "82ce157041ef1b9a4708aa321e944c41"
server_key_version: "encryptionkey@0"
I20260806 23:52:13.979477 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "734107795ec84e428301526718949b13"
format_stamp: "Formatted at 2026-08-06 23:52:13 on dist-test-slave-nj21"
server_key: "1d94f990e6d66e1619755eee678e9a17"
server_key_iv: "82ce157041ef1b9a4708aa321e944c41"
server_key_version: "encryptionkey@0"
I20260806 23:52:13.984835 20127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 23:52:13.988380 20148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:13.990150 20127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:52:13.990309 20127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "734107795ec84e428301526718949b13"
format_stamp: "Formatted at 2026-08-06 23:52:13 on dist-test-slave-nj21"
server_key: "1d94f990e6d66e1619755eee678e9a17"
server_key_iv: "82ce157041ef1b9a4708aa321e944c41"
server_key_version: "encryptionkey@0"
I20260806 23:52:13.990468 20127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:14.011554 20127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:14.015861 20127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:14.016074 20127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:14.016965 20127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:14.018077 20127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:14.018131 20127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:14.018182 20127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:14.018196 20127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:14.028170 20127 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:43457
I20260806 23:52:14.028215 20308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:43457 every 8 connection(s)
I20260806 23:52:14.029394 20127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:14.030515 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 20127
I20260806 23:52:14.030613 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:14.030896 11142 external_mini_cluster.cc:1428] Setting key 37bed3baccfc443c335f74c44da4b03d
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060333, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:14.053915 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:14.036913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:34973 (local address 127.10.225.190:46589)
0806 23:52:14.037047 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:14.037051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:14.038238 (+ 1187us) server_negotiation.cc:407] Connection header received
0806 23:52:14.040167 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:14.040172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:14.040260 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:14.040394 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:14.042152 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.042733 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.044582 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.044817 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.044925 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:14.050015 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:14.050020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:14.050022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:14.050054 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:14.051647 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.052265 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.052268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.052269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.052364 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.052719 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.052723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.052725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.052935 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:14.053022 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:14.053578 (+ 556us) server_negotiation.cc:299] Negotiation successful
0806 23:52:14.053715 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:52:14.054780 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:14.037488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46589 (local address 127.10.225.130:34973)
0806 23:52:14.038073 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:14.038120 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:14.039805 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:14.040619 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:14.040628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:14.041151 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:14.041948 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:14.041970 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.042881 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.042885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:14.044396 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:14.044409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.044993 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.044998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:14.045212 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:14.045877 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:14.045899 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:14.049723 (+ 3824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:14.051834 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:14.051841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:14.051857 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:14.052151 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:14.052456 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:14.052459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:14.052462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:14.052616 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:14.053039 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:14.053046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:14.053323 (+ 277us) client_negotiation.cc:769] Sending connection context
0806 23:52:14.053502 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 23:52:14.053889 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":163,"threads_started":1}
I20260806 23:52:14.056188 20309 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46589
I20260806 23:52:14.056493 20309 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:14.057178 20309 heartbeater.cc:511] Master 127.10.225.190:46589 requested a full tablet report, sending...
I20260806 23:52:14.058897 19823 ts_manager.cc:194] Registered new tserver with Master: 734107795ec84e428301526718949b13 (127.10.225.130:43457)
I20260806 23:52:14.059561 19823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:34973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:52:14.067003 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46589
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:14.193262 20318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:14.193578 20318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:14.193642 20318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:14.197989 20318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:14.198068 20318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:14.198158 20318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:14.203781 20318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:14.204968 20318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:14.206020 20318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:52:14.218765 20318 server_base.cc:1034] running on GCE node
W20260806 23:52:14.218708 20328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:14.218708 20326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:14.218708 20325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:14.219444 20318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:14.220184 20318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:14.221454 20318 hybrid_clock.cc:648] HybridClock initialized: now 1786060334221422 us; error 31 us; skew 500 ppm
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:14.224964 20318 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:52:14.226379 20318 webserver.cc:460] Webserver started at http://127.10.225.131:43935/ using document root <none> and password file <none>
I20260806 23:52:14.227023 20318 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:14.227102 20318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:14.227334 20318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:14.229667 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "6d7f4cc1fc4b446b848fcfee7d4f27be"
format_stamp: "Formatted at 2026-08-06 23:52:14 on dist-test-slave-nj21"
server_key: "81a5bb715c923e24ad3fa4cbeb0e2bb7"
server_key_iv: "53a8c7d0728979b75342558df0b00def"
server_key_version: "encryptionkey@0"
I20260806 23:52:14.230294 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "6d7f4cc1fc4b446b848fcfee7d4f27be"
format_stamp: "Formatted at 2026-08-06 23:52:14 on dist-test-slave-nj21"
server_key: "81a5bb715c923e24ad3fa4cbeb0e2bb7"
server_key_iv: "53a8c7d0728979b75342558df0b00def"
server_key_version: "encryptionkey@0"
I20260806 23:52:14.234715 20318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:14.238142 20338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:14.240170 20318 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:52:14.240336 20318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "6d7f4cc1fc4b446b848fcfee7d4f27be"
format_stamp: "Formatted at 2026-08-06 23:52:14 on dist-test-slave-nj21"
server_key: "81a5bb715c923e24ad3fa4cbeb0e2bb7"
server_key_iv: "53a8c7d0728979b75342558df0b00def"
server_key_version: "encryptionkey@0"
I20260806 23:52:14.240504 20318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:14.258360 20318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:14.261909 20318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:14.262131 20318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:14.262804 20318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:14.263799 20318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:14.263856 20318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:14.263919 20318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:14.263947 20318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:14.274861 20318 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:38551
I20260806 23:52:14.274879 20517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:38551 every 8 connection(s)
I20260806 23:52:14.275895 20318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:14.285888 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 20318
I20260806 23:52:14.286082 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:14.286401 11142 external_mini_cluster.cc:1428] Setting key ab8f915b76b8140e87158ee1c124019d
I20260806 23:52:14.291087 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:14.277971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:48923 (local address 127.10.225.190:46589)
0806 23:52:14.278142 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:14.278146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:14.278916 (+ 770us) server_negotiation.cc:407] Connection header received
0806 23:52:14.280339 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:14.280342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:14.280405 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:14.280478 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:14.281721 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.282251 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.283425 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.283640 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.283783 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:14.287185 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:14.287190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:14.287193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:14.287226 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:14.288787 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.289403 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.289408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.289409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.289511 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.289872 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.289877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.289879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.290092 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:14.290189 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:14.290762 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 23:52:14.290903 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 23:52:14.292019 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:14.278237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46589 (local address 127.10.225.131:48923)
0806 23:52:14.278753 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:14.278796 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:14.280133 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:14.280658 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:14.280665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:14.281028 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:14.281553 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:14.281567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.282370 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.282374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:14.283306 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:14.283316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.283835 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.283840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:14.284003 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:14.284723 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:14.284749 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:14.286912 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:14.288920 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:14.288928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:14.288948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:14.289295 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:14.289605 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:14.289609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:14.289611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:14.289774 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:14.290203 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:14.290221 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:14.290500 (+ 279us) client_negotiation.cc:769] Sending connection context
0806 23:52:14.290692 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 23:52:14.291072 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":134,"threads_started":1}
I20260806 23:52:14.293419 20518 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46589
I20260806 23:52:14.293684 20518 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:14.294234 20518 heartbeater.cc:511] Master 127.10.225.190:46589 requested a full tablet report, sending...
I20260806 23:52:14.295768 19823 ts_manager.cc:194] Registered new tserver with Master: 6d7f4cc1fc4b446b848fcfee7d4f27be (127.10.225.131:38551)
I20260806 23:52:14.296453 19823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:48923
I20260806 23:52:14.301079 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/test-admin.keytab.
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:14.320820 11142 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 23:52:14.324206 20526 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:14.322345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.10.225.129:45041)
0806 23:52:14.322673 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:14.322681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:14.322705 (+ 24us) server_negotiation.cc:407] Connection header received
0806 23:52:14.322758 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:14.322763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:14.322957 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:14.323082 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:14.323941 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46588: BlockingRecv error: recv got EOF from 127.0.0.1:46588 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":112,"threads_started":1}
W20260806 23:52:14.324291 11142 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5cc' of type FILE
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:14.325605 11142 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:14.326679 11142 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:52:14.337323 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:14.327652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.10.225.129:45041)
0806 23:52:14.327854 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:14.327858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:14.327876 (+ 18us) server_negotiation.cc:407] Connection header received
0806 23:52:14.327924 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:14.327927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:14.327974 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:14.328050 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:14.328575 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.329389 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.330231 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.330629 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.330719 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:14.333609 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:14.333620 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:14.333636 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:14.333671 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:14.335321 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.335883 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.335888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.335890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.336017 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.336292 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.336295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.336296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.336562 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:14.336764 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:14.337000 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 23:52:14.337164 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 23:52:14.341480 20526 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:14.340419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.10.225.129:45041)
0806 23:52:14.340545 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:14.340549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:14.340635 (+ 86us) server_negotiation.cc:407] Connection header received
0806 23:52:14.340762 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:14.340765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:14.340809 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:14.340866 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:14.341362 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46612: BlockingRecv error: recv got EOF from 127.0.0.1:46612 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 23:52:14.341552 11142 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5cc' of type FILE
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:14.342576 11142 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:14.343426 11142 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:52:14.353746 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:14.344328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.10.225.129:45041)
0806 23:52:14.344459 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:14.344463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:14.344585 (+ 122us) server_negotiation.cc:407] Connection header received
0806 23:52:14.344694 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:14.344697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:14.344737 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:14.344805 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:14.345354 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.345937 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.346707 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.346928 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.347014 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:14.349816 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:14.349821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:14.349824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:14.349867 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:14.351574 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.352241 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.352245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.352246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.352387 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.352761 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.352764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.352765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.352985 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:14.353128 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:14.353435 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 23:52:14.353562 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 23:52:14.356696 20526 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:14.355756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.10.225.129:45041)
0806 23:52:14.355868 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:14.355871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:14.355953 (+ 82us) server_negotiation.cc:407] Connection header received
0806 23:52:14.356060 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:14.356062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:14.356100 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:14.356163 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:14.356574 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46626: BlockingRecv error: recv got EOF from 127.0.0.1:46626 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 23:52:14.356878 11142 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786060299689793-11142-0/krb5kdc/krb5cc' of type FILE
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:14.357928 11142 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:14.358886 11142 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[19759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:52:14.369098 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:14.359818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.10.225.129:45041)
0806 23:52:14.359953 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:14.359956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:14.360028 (+ 72us) server_negotiation.cc:407] Connection header received
0806 23:52:14.360119 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:14.360122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:14.360159 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:14.360223 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:14.360654 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.361137 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.362003 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:14.362215 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:14.362306 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:14.365332 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:14.365337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:14.365340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:14.365365 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:14.366941 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.367471 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.367475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.367476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.367570 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:14.367948 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:14.367952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:14.367953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:14.368219 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:14.368374 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:14.368683 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 23:52:14.368815 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 23:52:14.370544 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 19916
I20260806 23:52:14.394608 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 20127
I20260806 23:52:14.403952 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 20318
I20260806 23:52:14.427312 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 19778
2026-08-06T23:52:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2472 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[20540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[20540](info): set up 2 sockets
Aug 06 23:52:14 dist-test-slave-nj21 krb5kdc[20540](info): commencing operation
krb5kdc: starting...
W20260806 23:52:15.595326 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:15 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.190.keytab.
Entry for principal oryx/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.190.keytab.
I20260806 23:52:15.737151 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:46319
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.190.keytab
--principal=oryx/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:46319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:15.863307 20559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:15.863617 20559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:15.863660 20559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:15.867928 20559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:15.868007 20559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:15.868027 20559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:15.868042 20559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:15.868057 20559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:15.874235 20559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:46319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.190.keytab
--principal=oryx/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:46319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:15.875490 20559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:15.876590 20559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:15.886179 20567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:15.886299 20569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:15.886240 20571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:15.886608 20559 server_base.cc:1034] running on GCE node
I20260806 23:52:15.887418 20559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:15.889207 20559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:15.890448 20559 hybrid_clock.cc:648] HybridClock initialized: now 1786060335890424 us; error 45 us; skew 500 ppm
Aug 06 23:52:15 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:15.893476 20559 init.cc:377] Logged in from keytab as oryx/127.10.225.190@KRBTEST.COM (short username oryx)
I20260806 23:52:15.894502 20559 webserver.cc:460] Webserver started at http://127.10.225.190:39341/ using document root <none> and password file <none>
I20260806 23:52:15.895097 20559 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:15.895169 20559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:15.895387 20559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:15.897626 20559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "8398b82fa36145bba7638367222ecd03"
format_stamp: "Formatted at 2026-08-06 23:52:15 on dist-test-slave-nj21"
server_key: "33a02fcafc9f3a1d4fb4bf09c7fe8c9e"
server_key_iv: "11a23b5ebfd24aead2fb1e1cec6f8da2"
server_key_version: "encryptionkey@0"
I20260806 23:52:15.898201 20559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "8398b82fa36145bba7638367222ecd03"
format_stamp: "Formatted at 2026-08-06 23:52:15 on dist-test-slave-nj21"
server_key: "33a02fcafc9f3a1d4fb4bf09c7fe8c9e"
server_key_iv: "11a23b5ebfd24aead2fb1e1cec6f8da2"
server_key_version: "encryptionkey@0"
I20260806 23:52:15.902676 20559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:52:15.906446 20582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:15.908269 20559 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 23:52:15.908427 20559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "8398b82fa36145bba7638367222ecd03"
format_stamp: "Formatted at 2026-08-06 23:52:15 on dist-test-slave-nj21"
server_key: "33a02fcafc9f3a1d4fb4bf09c7fe8c9e"
server_key_iv: "11a23b5ebfd24aead2fb1e1cec6f8da2"
server_key_version: "encryptionkey@0"
I20260806 23:52:15.908566 20559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:15.938259 20559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:15.942848 20559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:15.943109 20559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:15.952099 20559 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:46319
I20260806 23:52:15.952397 20664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:46319 every 8 connection(s)
I20260806 23:52:15.953397 20559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:15.957031 20665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:15.957293 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 20559
I20260806 23:52:15.957448 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:15.957902 11142 external_mini_cluster.cc:1428] Setting key 198a05e0d6b51037659e9523edd4a6b4
I20260806 23:52:15.966621 20665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: Bootstrap starting.
I20260806 23:52:15.969676 20665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: Neither blocks nor log segments found. Creating new log.
Aug 06 23:52:15 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:15.970556 20665 log.cc:824] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:15.973459 20665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: No bootstrap required, opened a new log
I20260806 23:52:15.976588 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:15.959487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.10.225.190:46319)
0806 23:52:15.959996 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:15.960017 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:15.960064 (+ 47us) server_negotiation.cc:407] Connection header received
0806 23:52:15.961103 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:15.961146 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:15.961835 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:15.962336 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:15.963403 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:15.965101 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:15.966428 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:15.967200 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:15.967401 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:15.970396 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:15.970403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:15.970425 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:15.970466 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:15.972809 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:15.973477 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:15.973484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:15.973491 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:15.973640 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:15.973996 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:15.973999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:15.974001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:15.974600 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:15.974780 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:15.975128 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 23:52:15.975472 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":180,"threads_started":1}
I20260806 23:52:15.978237 20665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } }
I20260806 23:52:15.978487 20665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:15.978521 20665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8398b82fa36145bba7638367222ecd03, State: Initialized, Role: FOLLOWER
I20260806 23:52:15.978998 20665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } }
I20260806 23:52:15.979166 20665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:15.979233 20665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:15.979333 20665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:15.980436 20665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } }
I20260806 23:52:15.980782 20665 leader_election.cc:302] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8398b82fa36145bba7638367222ecd03; no voters:
I20260806 23:52:15.981109 20665 leader_election.cc:288] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:15.981271 20673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:15.981501 20673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 1 LEADER]: Becoming Leader. State: Replica: 8398b82fa36145bba7638367222ecd03, State: Running, Role: LEADER
I20260806 23:52:15.981905 20673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } }
I20260806 23:52:15.982414 20665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:15.983533 20677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8398b82fa36145bba7638367222ecd03. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } }
I20260806 23:52:15.983702 20677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:15.983884 20676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } }
I20260806 23:52:15.984098 20676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
W20260806 23:52:15.987704 20702 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:52:15.987780 20702 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:52:15.987880 20693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:15.988731 20693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:15.995062 20693 catalog_manager.cc:1347] Generated new cluster ID: 97d8fd91848547a78de6f9001ce17931
I20260806 23:52:15.995150 20693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:16.028590 20693 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:16.029666 20693 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:16.037570 20693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: Generated new TSK 0
I20260806 23:52:16.038281 20693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab.
Entry for principal oryx/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab.
I20260806 23:52:16.083091 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46319
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab
--principal=oryx/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:16.222385 20710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:16.222671 20710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:16.222718 20710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:16.227756 20710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:16.227886 20710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:16.227986 20710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:16.235085 20710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab
--principal=oryx/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:16.236475 20710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:16.238267 20710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:16.251374 20720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:16.251374 20718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:16.251343 20717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:16.251573 20710 server_base.cc:1034] running on GCE node
I20260806 23:52:16.252197 20710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:16.252964 20710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:16.254172 20710 hybrid_clock.cc:648] HybridClock initialized: now 1786060336254152 us; error 33 us; skew 500 ppm
Aug 06 23:52:16 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060336, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:16.257476 20710 init.cc:377] Logged in from keytab as oryx/127.10.225.129@KRBTEST.COM (short username oryx)
I20260806 23:52:16.258857 20710 webserver.cc:460] Webserver started at http://127.10.225.129:35813/ using document root <none> and password file <none>
I20260806 23:52:16.259500 20710 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:16.259575 20710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:16.259843 20710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:16.262073 20710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "07476883c4614a9c88ef41245bb4b35c"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "c9776a1b5a2c8ccb1982ccdaad35a930"
server_key_iv: "096cb64fa0c84c70d8a7642ffb8c7795"
server_key_version: "encryptionkey@0"
I20260806 23:52:16.262670 20710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "07476883c4614a9c88ef41245bb4b35c"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "c9776a1b5a2c8ccb1982ccdaad35a930"
server_key_iv: "096cb64fa0c84c70d8a7642ffb8c7795"
server_key_version: "encryptionkey@0"
I20260806 23:52:16.266919 20710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260806 23:52:16.269990 20728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:16.271343 20710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:16.271461 20710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "07476883c4614a9c88ef41245bb4b35c"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "c9776a1b5a2c8ccb1982ccdaad35a930"
server_key_iv: "096cb64fa0c84c70d8a7642ffb8c7795"
server_key_version: "encryptionkey@0"
I20260806 23:52:16.271570 20710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:16.288062 20710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:16.292197 20710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:16.292400 20710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:16.293046 20710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:16.293936 20710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:16.293985 20710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:16.294042 20710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:16.294066 20710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:16.303314 20710 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:33373
I20260806 23:52:16.303363 20911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:33373 every 8 connection(s)
I20260806 23:52:16.304312 20710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:16.313830 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 20710
I20260806 23:52:16.313949 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:16.314234 11142 external_mini_cluster.cc:1428] Setting key e35d40317006a6e133a8e6f0871f831a
Aug 06 23:52:16 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060336, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:16.320497 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:16.306598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:48955 (local address 127.10.225.190:46319)
0806 23:52:16.306794 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:16.306798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:16.308036 (+ 1238us) server_negotiation.cc:407] Connection header received
0806 23:52:16.309485 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:16.309488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:16.309550 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:16.309630 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:16.310918 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.311463 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.312695 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.312955 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.313045 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:16.315775 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:16.315778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:16.315781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:16.315807 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:16.317661 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:16.318521 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:16.318526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:16.318527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:16.318728 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:16.319203 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:16.319207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:16.319209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:16.319461 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:16.319559 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:16.320228 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 23:52:16.320361 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 23:52:16.321714 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:16.307129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.129:48955)
0806 23:52:16.307862 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:16.307916 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:16.309245 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:16.309838 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:16.309845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:16.310235 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:16.310767 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:16.310781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.311586 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.311589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:16.312583 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:16.312594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.313113 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.313118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:16.313282 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:16.313863 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:16.313881 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:16.315571 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:16.317883 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:16.317896 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:16.317923 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:16.318349 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:16.318879 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:16.318883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:16.318885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:16.319081 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:16.319581 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:16.319588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:16.319948 (+ 360us) client_negotiation.cc:769] Sending connection context
0806 23:52:16.320157 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 23:52:16.320542 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":192,"threads_started":1}
I20260806 23:52:16.323225 20913 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46319
I20260806 23:52:16.323520 20913 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:16.324187 20913 heartbeater.cc:511] Master 127.10.225.190:46319 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:52:16.326928 20607 ts_manager.cc:194] Registered new tserver with Master: 07476883c4614a9c88ef41245bb4b35c (127.10.225.129:33373)
I20260806 23:52:16.328445 20607 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:48955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab.
Entry for principal oryx/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab.
I20260806 23:52:16.355104 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46319
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab
--principal=oryx/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:16.484346 20918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:16.484632 20918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:16.484686 20918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:16.488982 20918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:16.489063 20918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:16.489126 20918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:16.495889 20918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab
--principal=oryx/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:16.497448 20918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:16.498623 20918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:16.510071 20925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:16.510071 20935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:16.510071 20932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:16.510318 20918 server_base.cc:1034] running on GCE node
I20260806 23:52:16.510987 20918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:16.511740 20918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:16.512924 20918 hybrid_clock.cc:648] HybridClock initialized: now 1786060336512876 us; error 60 us; skew 500 ppm
Aug 06 23:52:16 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060336, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:16.516582 20918 init.cc:377] Logged in from keytab as oryx/127.10.225.130@KRBTEST.COM (short username oryx)
I20260806 23:52:16.517607 20918 webserver.cc:460] Webserver started at http://127.10.225.130:33171/ using document root <none> and password file <none>
I20260806 23:52:16.518201 20918 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:16.518273 20918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:16.518483 20918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:16.520750 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "f8b80aa9e9c647a1af02c6718b38d4a3"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "b5b7fceccbf8b506fd339261caccf2ba"
server_key_iv: "bc6150840de8f7263f0575a5c02d4045"
server_key_version: "encryptionkey@0"
I20260806 23:52:16.521347 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "f8b80aa9e9c647a1af02c6718b38d4a3"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "b5b7fceccbf8b506fd339261caccf2ba"
server_key_iv: "bc6150840de8f7263f0575a5c02d4045"
server_key_version: "encryptionkey@0"
I20260806 23:52:16.525338 20918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:52:16.528712 20943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:16.530180 20918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:52:16.530318 20918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "f8b80aa9e9c647a1af02c6718b38d4a3"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "b5b7fceccbf8b506fd339261caccf2ba"
server_key_iv: "bc6150840de8f7263f0575a5c02d4045"
server_key_version: "encryptionkey@0"
I20260806 23:52:16.530445 20918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:16.546343 20918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:16.551355 20918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:16.551578 20918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:16.552274 20918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:16.553171 20918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:16.553220 20918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:16.553277 20918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:16.553301 20918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:16.563277 20918 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:34931
I20260806 23:52:16.563351 21127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:34931 every 8 connection(s)
I20260806 23:52:16.564329 20918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:16.574240 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 20918
I20260806 23:52:16.574489 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:16.574851 11142 external_mini_cluster.cc:1428] Setting key 9f9dd6c6e1d29f2cd719b84be0e6d890
Aug 06 23:52:16 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060336, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:16.580718 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:16.566491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:58485 (local address 127.10.225.190:46319)
0806 23:52:16.566666 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:16.566671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:16.567791 (+ 1120us) server_negotiation.cc:407] Connection header received
0806 23:52:16.569481 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:16.569486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:16.569554 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:16.569651 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:16.571129 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.571819 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.573085 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.573440 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.573541 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:16.576992 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:16.576997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:16.577000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:16.577031 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:16.578471 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:16.579106 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:16.579110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:16.579112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:16.579210 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:16.579525 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:16.579528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:16.579530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:16.579812 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:16.579912 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:16.580416 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 23:52:16.580554 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 23:52:16.581571 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:16.566917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.130:58485)
0806 23:52:16.567565 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:16.567632 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:16.569172 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:16.569944 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:16.569953 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:16.570362 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:16.570918 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:16.570936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.571970 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.571973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:16.572921 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:16.572933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.573620 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.573624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:16.573780 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:16.574463 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:16.574486 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:16.576797 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:16.578596 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:16.578603 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:16.578621 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:16.578985 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:16.579300 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:16.579302 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:16.579303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:16.579448 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:16.579903 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:16.579910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:16.580175 (+ 265us) client_negotiation.cc:769] Sending connection context
0806 23:52:16.580345 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 23:52:16.580647 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":128,"threads_started":1}
I20260806 23:52:16.582759 21128 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46319
I20260806 23:52:16.583036 21128 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:16.583637 21128 heartbeater.cc:511] Master 127.10.225.190:46319 requested a full tablet report, sending...
I20260806 23:52:16.585373 20607 ts_manager.cc:194] Registered new tserver with Master: f8b80aa9e9c647a1af02c6718b38d4a3 (127.10.225.130:34931)
WARNING: no policy specified for oryx/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:52:16.586169 20607 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:58485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab.
Entry for principal oryx/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab.
I20260806 23:52:16.607722 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46319
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab
--principal=oryx/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:16.725440 21136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:16.725721 21136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:16.725785 21136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:16.729882 21136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:16.729957 21136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:16.730042 21136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:16.735608 21136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab
--principal=oryx/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:16.736820 21136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:16.737890 21136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:16.747767 21143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:16.747876 21136 server_base.cc:1034] running on GCE node
W20260806 23:52:16.747835 21146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:16.747767 21142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:16.748477 21136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:16.749123 21136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:16.750304 21136 hybrid_clock.cc:648] HybridClock initialized: now 1786060336750284 us; error 32 us; skew 500 ppm
Aug 06 23:52:16 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060336, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:16.753620 21136 init.cc:377] Logged in from keytab as oryx/127.10.225.131@KRBTEST.COM (short username oryx)
I20260806 23:52:16.754954 21136 webserver.cc:460] Webserver started at http://127.10.225.131:39789/ using document root <none> and password file <none>
I20260806 23:52:16.755738 21136 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:16.755817 21136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:16.756063 21136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:16.758683 21136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "b9bf272e79cc447db72a89c8e52aa8f9"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "86aa4cd650d315dd7b3a1c75f4987da1"
server_key_iv: "aa2728c433fbc656732ac2ab8aa20476"
server_key_version: "encryptionkey@0"
I20260806 23:52:16.759316 21136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "b9bf272e79cc447db72a89c8e52aa8f9"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "86aa4cd650d315dd7b3a1c75f4987da1"
server_key_iv: "aa2728c433fbc656732ac2ab8aa20476"
server_key_version: "encryptionkey@0"
I20260806 23:52:16.764240 21136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 23:52:16.767717 21153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:16.769418 21136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:52:16.769574 21136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "b9bf272e79cc447db72a89c8e52aa8f9"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "86aa4cd650d315dd7b3a1c75f4987da1"
server_key_iv: "aa2728c433fbc656732ac2ab8aa20476"
server_key_version: "encryptionkey@0"
I20260806 23:52:16.769722 21136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:16.782747 21136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:16.786912 21136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:16.787139 21136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:16.787868 21136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:16.788782 21136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:16.788828 21136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:16.788872 21136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:16.788900 21136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:16.797892 21136 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:44527
I20260806 23:52:16.797923 21325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:44527 every 8 connection(s)
I20260806 23:52:16.798833 21136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:16.805754 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 21136
I20260806 23:52:16.805902 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:16.806200 11142 external_mini_cluster.cc:1428] Setting key ac8066fc7af93ff75110365fdeb2578b
Aug 06 23:52:16 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060336, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:16.814915 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:16.800827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:58409 (local address 127.10.225.190:46319)
0806 23:52:16.801010 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:16.801015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:16.801873 (+ 858us) server_negotiation.cc:407] Connection header received
0806 23:52:16.803351 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:16.803355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:16.803442 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:16.803566 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:16.805544 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.806371 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.808006 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.808221 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.808298 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:16.811023 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:16.811026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:16.811029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:16.811054 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:16.812435 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:16.813026 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:16.813031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:16.813032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:16.813209 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:16.813654 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:16.813658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:16.813659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:16.813893 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:16.813985 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:16.814575 (+ 590us) server_negotiation.cc:299] Negotiation successful
0806 23:52:16.814750 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:52:16.815744 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:16.801187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.131:58409)
0806 23:52:16.801715 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:16.801760 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:16.803027 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:16.803979 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:16.803991 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:16.804535 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:16.805242 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:16.805292 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.806549 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.806554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:16.807875 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:16.807892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:16.808377 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:16.808383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:16.808592 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:16.809218 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:16.809237 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:16.810843 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:16.812562 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:16.812568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:16.812582 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:16.812896 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:16.813373 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:16.813375 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:16.813376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:16.813529 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:16.813993 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:16.813998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:16.814265 (+ 267us) client_negotiation.cc:769] Sending connection context
0806 23:52:16.814436 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 23:52:16.814724 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":129,"threads_started":1}
I20260806 23:52:16.817299 21329 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46319
I20260806 23:52:16.817754 21329 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:16.818454 21329 heartbeater.cc:511] Master 127.10.225.190:46319 requested a full tablet report, sending...
I20260806 23:52:16.820029 20606 ts_manager.cc:194] Registered new tserver with Master: b9bf272e79cc447db72a89c8e52aa8f9 (127.10.225.131:44527)
I20260806 23:52:16.820814 20606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:58409
I20260806 23:52:16.820924 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.189.keytab.
Entry for principal oryx/127.10.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.189.keytab.
I20260806 23:52:16.854580 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.189:39079
--webserver_interface=127.10.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.189.keytab
--principal=oryx/127.10.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:46319,127.10.225.189:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:16.979197 21335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:16.979530 21335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:16.979604 21335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:16.985008 21335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:16.985087 21335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:16.985114 21335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:16.985138 21335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:16.985170 21335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:16.991305 21335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:46319,127.10.225.189:39079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.189.keytab
--principal=oryx/127.10.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.189:39079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
W20260806 23:52:16.991603 21335 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.225.190:46319,127.10.225.189:39079'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 23:52:16.992619 21335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:16.993691 21335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:17.002628 21344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.002607 21342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:17.002730 21335 server_base.cc:1034] running on GCE node
W20260806 23:52:17.002607 21341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:17.003640 21335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:17.004911 21335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:17.006106 21335 hybrid_clock.cc:648] HybridClock initialized: now 1786060337006076 us; error 58 us; skew 500 ppm
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:17.009268 21335 init.cc:377] Logged in from keytab as oryx/127.10.225.189@KRBTEST.COM (short username oryx)
I20260806 23:52:17.010337 21335 webserver.cc:460] Webserver started at http://127.10.225.189:41435/ using document root <none> and password file <none>
I20260806 23:52:17.010952 21335 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:17.011027 21335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:17.011272 21335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:17.013728 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data/instance:
uuid: "967fd0ceedd74c40aa33e18b1723df9a"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9ff38f298df6ac2805888497dabeb736"
server_key_iv: "31ac3ce040e1dad5ab5fca006b254e1b"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.014364 21335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal/instance:
uuid: "967fd0ceedd74c40aa33e18b1723df9a"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9ff38f298df6ac2805888497dabeb736"
server_key_iv: "31ac3ce040e1dad5ab5fca006b254e1b"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.018555 21335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 23:52:17.021399 21355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.023123 21335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:52:17.023263 21335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal
uuid: "967fd0ceedd74c40aa33e18b1723df9a"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9ff38f298df6ac2805888497dabeb736"
server_key_iv: "31ac3ce040e1dad5ab5fca006b254e1b"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.023373 21335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:17.064908 21335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:17.069119 21335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:17.069377 21335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:17.077857 21335 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.189:39079
I20260806 23:52:17.078136 21438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.189:39079 every 8 connection(s)
I20260806 23:52:17.078912 21335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
I20260806 23:52:17.082201 21439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:17.084046 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 21335
I20260806 23:52:17.086794 21439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.225.189@KRBTEST.COM
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.189@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:17.099414 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.088689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.10.225.190:46319)
0806 23:52:17.088823 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.088828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.089253 (+ 425us) server_negotiation.cc:407] Connection header received
0806 23:52:17.089467 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.089471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.089528 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.089614 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.090677 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.091283 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.092358 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.092583 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.092676 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.095460 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.095465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.095467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.095502 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.097010 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.097541 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.097545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.097547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.097673 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.098015 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.098019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.098021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.098372 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.098486 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.098975 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.099188 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:52:17.100327 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.088657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53152)
0806 23:52:17.089125 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.089175 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.089340 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.089750 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.089757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.090192 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.090427 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.090440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.091424 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.091428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.092225 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.092235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.092743 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.092747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.092853 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.093414 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.093432 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.095260 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.097123 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.097133 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.097145 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.097445 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.097760 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.097764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.097765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.097911 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.098479 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.098485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.098650 (+ 165us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.098831 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.099218 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":185,"threads_started":1}
I20260806 23:52:17.100332 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.086121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.10.225.189:39079)
0806 23:52:17.086644 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.086664 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.086713 (+ 49us) server_negotiation.cc:407] Connection header received
0806 23:52:17.087621 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.087648 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.088079 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.088517 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.089303 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.090460 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.091339 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.091898 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.092046 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.094935 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.094941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.094952 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.094977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.096894 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.097430 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.097435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.097436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.097597 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.097925 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.097928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.097930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.098470 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.098631 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.098991 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.099313 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":340,"thread_start_us":142,"threads_started":1}
I20260806 23:52:17.102108 21439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.189@KRBTEST.COM for oryx/127.10.225.189@KRBTEST.COM
I20260806 23:52:17.111126 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.102672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.0.0.1:51542)
0806 23:52:17.102782 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.102795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.102876 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.103039 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.103041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.103178 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.103271 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.103275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.104071 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.104075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.104758 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.104769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.105320 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.105323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.105436 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.105916 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.105928 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.107227 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.109358 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.109362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.109364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.109596 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.109981 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.109984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.109986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.110097 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.110650 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.110653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.110747 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.110842 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.110986 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260806 23:52:17.111227 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.102798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.10.225.189:39079)
0806 23:52:17.102893 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.102896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.102908 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:52:17.102936 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.102938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.102971 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.103030 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.103367 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.103927 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.104901 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.105202 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.105312 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.107411 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.107415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.107418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.107444 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.109203 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.109743 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.109747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.109748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.109837 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.110235 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.110239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.110240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.110540 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.110630 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.110967 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.111077 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":35}
I20260806 23:52:17.114957 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Bootstrap starting.
I20260806 23:52:17.117349 21439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:17.118057 21439 log.cc:824] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:17.120813 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: No bootstrap required, opened a new log
I20260806 23:52:17.124600 21439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } }
I20260806 23:52:17.124864 21439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:17.124913 21439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967fd0ceedd74c40aa33e18b1723df9a, State: Initialized, Role: FOLLOWER
I20260806 23:52:17.125453 21439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } }
I20260806 23:52:17.126246 21447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } } }
I20260806 23:52:17.126469 21447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:52:17.127164 21439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: configured and running, proceeding with master startup.
W20260806 23:52:17.133446 21461 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:52:17.133559 21461 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:52:17.135888 21335 master_runner.cc:428] Detected that this master 967fd0ceedd74c40aa33e18b1723df9a is joining an existing cluster
I20260806 23:52:17.135963 21335 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.225.189:39079
I20260806 23:52:17.147580 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.138654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.0.0.1:51544)
0806 23:52:17.139295 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.139313 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.139470 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.139882 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.139886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.140359 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.140527 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.140537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.141585 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.141588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.142488 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.142499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.143116 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.143120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.143211 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.143636 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.143653 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.144205 (+ 552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.146016 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.146019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.146020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.146274 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.146532 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.146535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.146537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.146610 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.146990 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.146993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.147104 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.147205 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.147360 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"spinlock_wait_cycles":4864,"thread_start_us":186,"threads_started":1}
I20260806 23:52:17.147814 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.138878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.10.225.189:39079)
0806 23:52:17.138991 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.138996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.139464 (+ 468us) server_negotiation.cc:407] Connection header received
0806 23:52:17.139572 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.139575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.139632 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.139754 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.140654 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.141451 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.142604 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.142920 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.142988 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.144338 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.144342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.144344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.144366 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.145930 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.146384 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.146387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.146389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.146457 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.146684 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.146687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.146689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.146897 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.146998 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.147264 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.147625 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 23:52:17.147921 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.138764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.10.225.190:46319)
0806 23:52:17.139005 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.139009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.139424 (+ 415us) server_negotiation.cc:407] Connection header received
0806 23:52:17.139705 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.139709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.139778 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.139856 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.140639 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.141339 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.142261 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.142509 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.142601 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.144313 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.144318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.144321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.144344 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.145840 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.146360 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.146363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.146365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.146456 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.146730 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.146734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.146735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.146950 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.147032 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.147472 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.147761 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260806 23:52:17.148474 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.138654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53164)
0806 23:52:17.139295 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.139313 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.139470 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.139907 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.139911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.140332 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.140512 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.140519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.141498 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.141501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.142137 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.142146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.142632 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.142635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.142730 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.143719 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.143733 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.144172 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.145951 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.145954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.145956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.146257 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.146563 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.146566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.146568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.146654 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.147072 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.147075 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.147152 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.148151 (+ 999us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.148308 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":32,"spinlock_wait_cycles":57088,"thread_start_us":138,"threads_started":1}
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.190@KRBTEST.COM for oryx/127.10.225.189@KRBTEST.COM
I20260806 23:52:17.164306 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.154507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.0.0.1:51550)
0806 23:52:17.155049 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.155127 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.155286 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.156089 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.156096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.156618 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.156774 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.156783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.157631 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.157635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.158286 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.158296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.158728 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.158732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.158839 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.159378 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.159391 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.160898 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.162646 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.162657 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.162674 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.162909 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.163165 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.163168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.163170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.163250 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.163616 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.163624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.163790 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.163927 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.164131 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":158,"threads_started":1}
I20260806 23:52:17.164304 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.155636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.10.225.189:39079)
0806 23:52:17.155776 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.155781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.155794 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:52:17.155868 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.155871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.155922 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.156012 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.156904 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.157516 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.158409 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.158623 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.158708 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.161035 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.161039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.161040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.161063 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.162518 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.163005 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.163008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.163009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.163083 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.163331 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.163334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.163336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.163517 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.163602 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.163997 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.164119 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for oryx/127.10.225.188@KRBTEST.COM; defaulting to no policy
I20260806 23:52:17.166174 20606 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.225.189:39079: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a"
member_type: NON_VOTER
last_known_addr {
host: "127.10.225.189"
port: 39079
}
attrs {
promote: true
}
}
dest_uuid: "8398b82fa36145bba7638367222ecd03"
cas_config_opid_index: -1
I20260806 23:52:17.166916 20606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.188@KRBTEST.COM" created.
I20260806 23:52:17.170037 21394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:17.169258 21479 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 967fd0ceedd74c40aa33e18b1723df9a (127.10.225.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } } }
I20260806 23:52:17.172102 21394 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Refusing update from remote peer 8398b82fa36145bba7638367222ecd03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 23:52:17.173278 21479 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.225.189:39079
I20260806 23:52:17.173568 21479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } } }
I20260806 23:52:17.173669 21479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:17.173852 21335 master.cc:537] Master@127.10.225.189:39079 shutting down...
I20260806 23:52:17.173466 21477 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 23:52:17.177055 20585 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 23:52:17.178444 20585 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 -> Peer 967fd0ceedd74c40aa33e18b1723df9a (127.10.225.189:39079): Couldn't send request to peer 967fd0ceedd74c40aa33e18b1723df9a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 23:52:17.179014 21335 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:17.179315 21335 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:17.179431 21335 tablet_replica.cc:330] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.188.keytab.
Entry for principal oryx/127.10.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.188.keytab.
I20260806 23:52:17.189572 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.188:38663
--webserver_interface=127.10.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.188.keytab
--principal=oryx/127.10.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:46319,127.10.225.189:39079,127.10.225.188:38663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:52:17.195137 21335 master.cc:552] Master@127.10.225.189:39079 shutdown complete.
I20260806 23:52:17.195227 21335 master_runner.cc:305] Clearing existing system tablet
I20260806 23:52:17.195528 21335 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:17.195739 21335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:17.197407 21335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:17.198343 21488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.198629 21335 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:17.198746 21335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:17.198779 21335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal
uuid: "967fd0ceedd74c40aa33e18b1723df9a"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9ff38f298df6ac2805888497dabeb736"
server_key_iv: "31ac3ce040e1dad5ab5fca006b254e1b"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.200889 21335 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:17.202827 21335 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 23:52:17.202929 21335 log.cc:1196] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 23:52:17.203171 21335 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Deleting consensus metadata
I20260806 23:52:17.203336 21335 master_runner.cc:315] Copying system tablet from 127.10.225.190:46319
I20260806 23:52:17.204691 21335 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: tablet copy: Beginning tablet copy session from remote peer at address 127.10.225.190:46319
I20260806 23:52:17.212585 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.205398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.10.225.190:46319)
0806 23:52:17.205539 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.205544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.205747 (+ 203us) server_negotiation.cc:407] Connection header received
0806 23:52:17.205873 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.205876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.205928 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.205994 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.206460 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.207002 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.208018 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.208251 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.208349 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.209437 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.209441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.209443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.209481 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.210754 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.211291 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.211294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.211295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.211379 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.211651 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.211654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.211656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.211902 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.211992 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.212303 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.212424 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:52:17.212667 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.205297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53176)
0806 23:52:17.205656 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.205667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.205783 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.206022 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.206025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.206223 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.206342 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.206348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.207105 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.207109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.207902 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.207914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.208357 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.208361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.208461 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.208951 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.208966 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.209295 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.210865 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.210868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.210870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.211171 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.211478 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.211481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.211483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.211568 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.211995 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.211998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.212098 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.212242 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.212406 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":95,"threads_started":1}
I20260806 23:52:17.214008 20640 tablet_copy_service.cc:140] P 8398b82fa36145bba7638367222ecd03: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 967fd0ceedd74c40aa33e18b1723df9a ({username='oryx', principal='oryx/127.10.225.189@KRBTEST.COM'} at 127.0.0.1:53176)
I20260806 23:52:17.214294 20640 tablet_copy_service.cc:161] P 8398b82fa36145bba7638367222ecd03: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 967fd0ceedd74c40aa33e18b1723df9a at {username='oryx', principal='oryx/127.10.225.189@KRBTEST.COM'} at 127.0.0.1:53176: session id = 967fd0ceedd74c40aa33e18b1723df9a-00000000000000000000000000000000
I20260806 23:52:17.216701 20640 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 23:52:17.218645 21335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 23:52:17.220885 21335 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: tablet copy: Starting download of 0 data blocks...
I20260806 23:52:17.221082 21335 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: tablet copy: Starting download of 1 WAL segments...
I20260806 23:52:17.222904 21335 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 23:52:17.224577 20640 tablet_copy_service.cc:342] P 8398b82fa36145bba7638367222ecd03: Request end of tablet copy session 967fd0ceedd74c40aa33e18b1723df9a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.225.189@KRBTEST.COM'} at 127.0.0.1:53176
I20260806 23:52:17.224694 20640 tablet_copy_service.cc:433] P 8398b82fa36145bba7638367222ecd03: ending tablet copy session 967fd0ceedd74c40aa33e18b1723df9a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 967fd0ceedd74c40aa33e18b1723df9a
I20260806 23:52:17.234421 21335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:17.236234 21505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.236260 21501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.236528 21503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:17.236855 21335 server_base.cc:1034] running on GCE node
I20260806 23:52:17.237015 21335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:17.237323 21335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:17.238483 21335 hybrid_clock.cc:648] HybridClock initialized: now 1786060337238465 us; error 32 us; skew 500 ppm
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:17.240015 21335 init.cc:377] Logged in from keytab as oryx/127.10.225.189@KRBTEST.COM (short username oryx)
I20260806 23:52:17.240597 21335 webserver.cc:460] Webserver started at http://127.10.225.189:43739/ using document root <none> and password file <none>
I20260806 23:52:17.240795 21335 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:17.240833 21335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:17.241918 21335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:17.242780 21515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.242995 21335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:17.243068 21335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal
uuid: "967fd0ceedd74c40aa33e18b1723df9a"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9ff38f298df6ac2805888497dabeb736"
server_key_iv: "31ac3ce040e1dad5ab5fca006b254e1b"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.243149 21335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:17.260104 21335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:17.261889 21335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:17.266760 21335 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.189:39079
I20260806 23:52:17.267355 21335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
I20260806 23:52:17.268989 21596 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:52:17.269862 21596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Bootstrap starting.
I20260806 23:52:17.272948 21595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.189:39079 every 8 connection(s)
I20260806 23:52:17.279362 21596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:52:17.279704 21596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Bootstrap complete.
I20260806 23:52:17.280361 21596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } }
I20260806 23:52:17.280480 21596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 967fd0ceedd74c40aa33e18b1723df9a, State: Initialized, Role: LEARNER
I20260806 23:52:17.280593 21596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } }
I20260806 23:52:17.280922 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } } }
I20260806 23:52:17.281093 21596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:17.281102 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: This master's current role is: LEARNER
I20260806 23:52:17.284113 21613 catalog_manager.cc:1259] Loaded cluster ID: 97d8fd91848547a78de6f9001ce17931
I20260806 23:52:17.284253 21613 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: loading cluster ID for follower catalog manager: success
I20260806 23:52:17.285892 21613 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: acquiring CA information for follower catalog manager: success
I20260806 23:52:17.287053 21613 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:52:17.331923 20913 heartbeater.cc:503] Master 127.10.225.190:46319 was elected leader, sending a full tablet report...
W20260806 23:52:17.356895 21486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:17.357179 21486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:17.357219 21486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:17.361203 21486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:17.361274 21486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:17.361291 21486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:17.361308 21486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:17.361322 21486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:17.366772 21486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:46319,127.10.225.189:39079,127.10.225.188:38663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.188.keytab
--principal=oryx/127.10.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.188:38663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:17.367885 21486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:17.368894 21486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:17.377559 21619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.377553 21620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.377594 21622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:17.377892 21486 server_base.cc:1034] running on GCE node
I20260806 23:52:17.378504 21486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:17.379642 21486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:17.380939 21486 hybrid_clock.cc:648] HybridClock initialized: now 1786060337380912 us; error 36 us; skew 500 ppm
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:17.384461 21486 init.cc:377] Logged in from keytab as oryx/127.10.225.188@KRBTEST.COM (short username oryx)
I20260806 23:52:17.385494 21486 webserver.cc:460] Webserver started at http://127.10.225.188:35057/ using document root <none> and password file <none>
I20260806 23:52:17.386092 21486 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:17.386159 21486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:17.386371 21486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:17.389330 21486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data/instance:
uuid: "42b3078ff45f478a98e40232ca8dad04"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9d8a0c3ca7e281c096ccef70e59be30f"
server_key_iv: "d04b85dfb87159478ac1d0708ec9c670"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.390287 21486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal/instance:
uuid: "42b3078ff45f478a98e40232ca8dad04"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9d8a0c3ca7e281c096ccef70e59be30f"
server_key_iv: "d04b85dfb87159478ac1d0708ec9c670"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.395049 21486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 23:52:17.398912 21633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.400404 21486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:17.400535 21486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal
uuid: "42b3078ff45f478a98e40232ca8dad04"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9d8a0c3ca7e281c096ccef70e59be30f"
server_key_iv: "d04b85dfb87159478ac1d0708ec9c670"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.400653 21486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:17.423478 21486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:17.427100 21486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:17.427275 21486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:17.436136 21486 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.188:38663
I20260806 23:52:17.436453 21728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.188:38663 every 8 connection(s)
I20260806 23:52:17.437183 21486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
I20260806 23:52:17.439841 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 21486
I20260806 23:52:17.441257 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:17.446534 21730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.225.188@KRBTEST.COM
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.188@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:17.455149 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.441840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.10.225.188:38663)
0806 23:52:17.442542 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.442558 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.442599 (+ 41us) server_negotiation.cc:407] Connection header received
0806 23:52:17.443312 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.443330 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.443789 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.444136 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.444990 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.446002 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.446842 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.447346 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.447519 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.450296 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.450303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.450313 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.450347 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.452003 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.452405 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.452410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.452414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.452526 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.452772 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.452775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.452777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.453362 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.453608 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.453889 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.454176 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"spinlock_wait_cycles":1280,"thread_start_us":294,"threads_started":1}
I20260806 23:52:17.458395 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.448655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53180)
0806 23:52:17.449248 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.449320 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.449553 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.450048 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.450056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.450436 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.450575 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.450583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.451430 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.451434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.452142 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.452153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.452583 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.452587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.452705 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.453215 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.453227 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.454741 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.456742 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.456746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.456750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.456960 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.457297 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.457300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.457301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.457385 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.457835 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.457841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.457961 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.458078 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.458227 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":189,"threads_started":1}
I20260806 23:52:17.458631 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.448662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.10.225.190:46319)
0806 23:52:17.448839 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.448844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.449743 (+ 899us) server_negotiation.cc:407] Connection header received
0806 23:52:17.449803 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.449805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.449860 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.449928 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.450732 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.451281 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.452261 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.452477 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.452565 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.454969 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.454974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.454976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.455017 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.456641 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.457057 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.457061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.457063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.457160 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.457469 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.457472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.457474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.457701 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.457805 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.458234 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.458415 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 23:52:17.459860 21730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.188@KRBTEST.COM for oryx/127.10.225.189@KRBTEST.COM
I20260806 23:52:17.468482 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.460637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.10.225.189:39079)
0806 23:52:17.460976 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.460980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.460993 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:52:17.461030 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.461033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.461075 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.461159 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.461552 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.462224 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.462974 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.463198 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.463286 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.465439 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.465442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.465444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.465468 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.466796 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.467193 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.467196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.467197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.467269 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.467582 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.467586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.467587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.467849 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.467941 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.468192 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.468318 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":132,"threads_started":1}
I20260806 23:52:17.468480 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.460396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.0.0.1:51564)
0806 23:52:17.460523 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.460535 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.460598 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.461188 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.461190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.461339 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.461442 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.461448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.462324 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.462327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.462869 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.462877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.463295 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.463299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.463395 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.463885 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.463896 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.465294 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.466914 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.466918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.466920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.467105 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.467352 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.467355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.467357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.467432 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.467942 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.467945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.468042 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.468128 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.468280 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 23:52:17.469388 21730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.188@KRBTEST.COM for oryx/127.10.225.188@KRBTEST.COM
I20260806 23:52:17.478318 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.469963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.10.225.188:38663)
0806 23:52:17.470084 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.470088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.470101 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:52:17.470130 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.470132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.470165 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.470232 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.470572 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.471110 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.472760 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.473009 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.473094 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.475442 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.475445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.475447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.475467 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.476738 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.477182 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.477186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.477187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.477269 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.477528 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.477531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.477533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.477711 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.477797 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.478027 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.478184 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":57}
I20260806 23:52:17.478318 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.469902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:38663 (local address 127.0.0.1:33950)
0806 23:52:17.469997 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.470012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.470080 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.470249 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.470251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.470382 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.470456 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.470460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.471880 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.471886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.472618 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.472631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.473137 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.473141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.473252 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.474013 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.474026 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.475315 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.476835 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.476838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.476841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.477021 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.477352 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.477355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.477357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.477450 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.477798 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.477801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.477885 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.477972 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.478126 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 23:52:17.482422 21730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: Bootstrap starting.
I20260806 23:52:17.485148 21730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:17.485929 21730 log.cc:824] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:17.488397 21730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: No bootstrap required, opened a new log
I20260806 23:52:17.491592 21730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } }
I20260806 23:52:17.491856 21730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:17.491899 21730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b3078ff45f478a98e40232ca8dad04, State: Initialized, Role: FOLLOWER
I20260806 23:52:17.492372 21730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } }
I20260806 23:52:17.492936 21741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } } }
I20260806 23:52:17.493083 21741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:52:17.493702 21730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 23:52:17.499214 21758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:52:17.499300 21758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:52:17.502600 21486 master_runner.cc:428] Detected that this master 42b3078ff45f478a98e40232ca8dad04 is joining an existing cluster
I20260806 23:52:17.502681 21486 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.225.188:38663
I20260806 23:52:17.509578 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 20710
I20260806 23:52:17.521245 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.507830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.10.225.188:38663)
0806 23:52:17.507985 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.507991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.509819 (+ 1828us) server_negotiation.cc:407] Connection header received
0806 23:52:17.509979 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.509982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.510024 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.510074 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.510714 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.511479 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.512760 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.513117 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.513196 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.515709 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.515715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.515718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.515742 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.519142 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.519816 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.519821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.519823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.519906 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.520157 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.520160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.520161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.520430 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.520506 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.520843 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.521022 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:52:17.521907 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.507720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:38663 (local address 127.0.0.1:33964)
0806 23:52:17.509720 (+ 2000us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.509738 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.509881 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.510205 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.510220 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.510446 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.510576 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.510585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.511620 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.511623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.512615 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.512629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.513448 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.513452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.513563 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.514868 (+ 1305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.514885 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.515500 (+ 615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.519369 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.519373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.519375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.519661 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.520005 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.520007 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.520009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.520083 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.520570 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.520572 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.520684 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.521558 (+ 874us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.521723 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1926,"mutex_wait_us":42,"spinlock_wait_cycles":768,"thread_start_us":348,"threads_started":1}
I20260806 23:52:17.533741 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.506866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53196)
0806 23:52:17.508541 (+ 1675us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.508560 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.508792 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.509294 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.509297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.525522 (+ 16225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.525695 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.525705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.526826 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.526829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.528144 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.528157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.528854 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.528857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.528957 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.529524 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.529539 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.530034 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.532002 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.532006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.532008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.532244 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.532568 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.532571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.532572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.532678 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.533120 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.533123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.533229 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.533332 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.533508 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1581,"mutex_wait_us":848,"thread_start_us":215,"threads_started":1}
I20260806 23:52:17.533779 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.507008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.10.225.190:46319)
0806 23:52:17.507207 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.507213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.508673 (+ 1460us) server_negotiation.cc:407] Connection header received
0806 23:52:17.508939 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.508943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.509013 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.509078 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.525910 (+ 16832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.526671 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.528284 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.528599 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.528673 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.530220 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.530225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.530228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.530256 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.531877 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.532375 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.532379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.532381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.532476 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.532766 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.532769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.532771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.533025 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.533119 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.533476 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.533609 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 23:52:17.535645 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:33373
--local_ip_for_outbound_sockets=127.10.225.129
--tserver_master_addrs=127.10.225.190:46319
--webserver_port=35813
--webserver_interface=127.10.225.129
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab
--principal=oryx/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.225.190:46319,127.10.225.189:39079
--tserver_master_addrs=127.10.225.190:46319,127.10.225.189:39079,127.10.225.188:38663 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060335, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.190@KRBTEST.COM for oryx/127.10.225.188@KRBTEST.COM
I20260806 23:52:17.551604 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.538528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:38663 (local address 127.0.0.1:33978)
0806 23:52:17.538726 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.538740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.538836 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.539145 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.539148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.539523 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.539779 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.539795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.541161 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.541165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.542134 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.542147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.543251 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.543256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.543388 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.543987 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.544002 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.546139 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.549840 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.549844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.549847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.550138 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.550456 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.550459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.550461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.550550 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.551019 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.551023 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.551137 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.551231 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.551414 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 23:52:17.552613 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.538586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33978 (local address 127.10.225.188:38663)
0806 23:52:17.538706 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.538711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.538802 (+ 91us) server_negotiation.cc:407] Connection header received
0806 23:52:17.538968 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.538971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.539028 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.539110 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.539956 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.540967 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.542301 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.542821 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.542997 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.546382 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.546387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.546390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.546431 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.549675 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.550264 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.550268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.550270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.550359 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.550640 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.550643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.550644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.550886 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.550974 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.552330 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.552459 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:52:17.553273 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.507279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.10.225.189:39079)
0806 23:52:17.507665 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.507699 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.508480 (+ 781us) server_negotiation.cc:407] Connection header received
0806 23:52:17.508722 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.508726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.508797 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.508853 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.526168 (+ 17315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.527012 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.528360 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.528716 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.528790 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.549767 (+ 20977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.549772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.549775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.549809 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.551447 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.551979 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.551983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.551985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.552069 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.552346 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.552349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.552351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.552570 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.552648 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.552977 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.553116 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260806 23:52:17.554061 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.506872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.0.0.1:51568)
0806 23:52:17.508381 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.508398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.508585 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.509060 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.509063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.509565 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.509724 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.509732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.527180 (+ 17448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.527183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.528217 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.528228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.528971 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.528974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.529070 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.548926 (+ 19856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.548963 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.549523 (+ 560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.551588 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.551591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.551593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.551859 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.552171 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.552173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.552175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.552255 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.552704 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.552707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.552796 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.553715 (+ 919us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.553872 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"mutex_wait_us":24,"thread_start_us":272,"threads_started":1}
I20260806 23:52:17.554631 20606 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.225.188:38663: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "42b3078ff45f478a98e40232ca8dad04"
member_type: NON_VOTER
last_known_addr {
host: "127.10.225.188"
port: 38663
}
attrs {
promote: true
}
}
dest_uuid: "8398b82fa36145bba7638367222ecd03"
cas_config_opid_index: 5
I20260806 23:52:17.555351 20606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } }
I20260806 23:52:17.556730 21479 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.558229 21479 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.225.188:38663
I20260806 23:52:17.558406 21479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.558501 21479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:17.559198 21486 master.cc:537] Master@127.10.225.188:38663 shutting down...
W20260806 23:52:17.562208 20587 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 -> Peer 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188:38663): Couldn't send request to peer 42b3078ff45f478a98e40232ca8dad04. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 23:52:17.565780 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.557344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.10.225.189:39079)
0806 23:52:17.557754 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.557758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.557927 (+ 169us) server_negotiation.cc:407] Connection header received
0806 23:52:17.558112 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.558115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.558177 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.558245 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:17.558636 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.559478 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.561242 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.562253 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.562442 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.562685 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.565490 (+ 2805us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.565635 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260806 23:52:17.566430 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.557070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.0.0.1:51584)
0806 23:52:17.557848 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.557862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.558000 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.558403 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.558406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.558424 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:17.558524 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.558529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.559617 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.559621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.561100 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.561112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.562857 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.562862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.565115 (+ 2253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.565198 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.566172 (+ 974us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.566290 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704}
I20260806 23:52:17.567852 21563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 LEARNER]: Refusing update from remote peer 8398b82fa36145bba7638367222ecd03: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 23:52:17.568444 21481 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 23:52:17.571153 21563 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.573704 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8398b82fa36145bba7638367222ecd03. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.573838 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: This master's current role is: LEARNER
I20260806 23:52:17.573932 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: NON_VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: true } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.574002 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: This master's current role is: LEARNER
I20260806 23:52:17.574630 21486 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:17.574973 21486 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:17.575129 21486 tablet_replica.cc:330] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: stopping tablet replica
I20260806 23:52:17.576529 21481 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: attempting to promote NON_VOTER 967fd0ceedd74c40aa33e18b1723df9a to VOTER
I20260806 23:52:17.577013 21481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } }
W20260806 23:52:17.578451 20587 connection.cc:535] client connection to 127.10.225.188:38663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 23:52:17.579283 21559 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 LEARNER]: Refusing update from remote peer 8398b82fa36145bba7638367222ecd03: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 23:52:17.579728 21477 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 23:52:17.580536 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.580190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:38663 (local address 127.0.0.1:33986)
0806 23:52:17.580352 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.580429 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:38663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 23:52:17.580767 20587 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 -> Peer 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188:38663): Couldn't send request to peer 42b3078ff45f478a98e40232ca8dad04. Status: Network error: Client connection negotiation failed: client connection to 127.10.225.188:38663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 23:52:17.582415 21486 master.cc:552] Master@127.10.225.188:38663 shutdown complete.
I20260806 23:52:17.582473 21486 master_runner.cc:305] Clearing existing system tablet
I20260806 23:52:17.582734 21486 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:17.582846 21486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:17.584594 21486 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:17.585371 21477 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 967fd0ceedd74c40aa33e18b1723df9a (127.10.225.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.589053 21559 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 967fd0ceedd74c40aa33e18b1723df9a (127.10.225.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.591100 21128 heartbeater.cc:503] Master 127.10.225.190:46319 was elected leader, sending a full tablet report...
I20260806 23:52:17.591455 21772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.591588 21772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:52:17.592247 21477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.592495 21477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:17.592466 21481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.592837 21481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:17.592933 21774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.593189 21486 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:17.593329 21486 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260806 23:52:17.593376 21486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal
uuid: "42b3078ff45f478a98e40232ca8dad04"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9d8a0c3ca7e281c096ccef70e59be30f"
server_key_iv: "d04b85dfb87159478ac1d0708ec9c670"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.595743 21486 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:17.598652 21486 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 23:52:17.598776 21486 log.cc:1196] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 23:52:17.599072 21486 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: Deleting consensus metadata
I20260806 23:52:17.599253 21486 master_runner.cc:315] Copying system tablet from 127.10.225.190:46319
I20260806 23:52:17.600769 21486 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: tablet copy: Beginning tablet copy session from remote peer at address 127.10.225.190:46319
I20260806 23:52:17.611378 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.601429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53206)
0806 23:52:17.601825 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.601841 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.601975 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.602289 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.602292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.602528 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.602686 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.602693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.603813 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.603817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.604676 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.604687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.605282 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.605286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.605387 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.606428 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.606442 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.606910 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.609391 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.609395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.609397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.609713 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.610059 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.610062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.610064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.610164 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.610769 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.610772 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.610902 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.611016 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.611182 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":99,"threads_started":1}
I20260806 23:52:17.611409 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.601713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.10.225.190:46319)
0806 23:52:17.601866 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.601871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.601924 (+ 53us) server_negotiation.cc:407] Connection header received
0806 23:52:17.602074 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.602077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.602131 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.602231 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.602815 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.603634 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.604817 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.605166 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.605268 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.607066 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.607070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.607074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.607102 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.609219 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.609841 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.609845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.609847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.609959 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.610341 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.610346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.610348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.610656 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.610774 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.611074 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.611230 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 23:52:17.612376 20640 tablet_copy_service.cc:140] P 8398b82fa36145bba7638367222ecd03: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 42b3078ff45f478a98e40232ca8dad04 ({username='oryx', principal='oryx/127.10.225.188@KRBTEST.COM'} at 127.0.0.1:53206)
I20260806 23:52:17.612493 20640 tablet_copy_service.cc:161] P 8398b82fa36145bba7638367222ecd03: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 42b3078ff45f478a98e40232ca8dad04 at {username='oryx', principal='oryx/127.10.225.188@KRBTEST.COM'} at 127.0.0.1:53206: session id = 42b3078ff45f478a98e40232ca8dad04-00000000000000000000000000000000
I20260806 23:52:17.613853 20640 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 23:52:17.615815 21486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 23:52:17.618517 21486 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: tablet copy: Starting download of 0 data blocks...
I20260806 23:52:17.618732 21486 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: tablet copy: Starting download of 1 WAL segments...
I20260806 23:52:17.620579 21486 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 23:52:17.622951 20640 tablet_copy_service.cc:342] P 8398b82fa36145bba7638367222ecd03: Request end of tablet copy session 42b3078ff45f478a98e40232ca8dad04-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.225.188@KRBTEST.COM'} at 127.0.0.1:53206
I20260806 23:52:17.623055 20640 tablet_copy_service.cc:433] P 8398b82fa36145bba7638367222ecd03: ending tablet copy session 42b3078ff45f478a98e40232ca8dad04-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 42b3078ff45f478a98e40232ca8dad04
I20260806 23:52:17.631646 21486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:17.633500 21785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.633577 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.633831 21786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:17.634083 21486 server_base.cc:1034] running on GCE node
I20260806 23:52:17.634281 21486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:17.634598 21486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:17.635732 21486 hybrid_clock.cc:648] HybridClock initialized: now 1786060337635720 us; error 30 us; skew 500 ppm
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:17.637526 21486 init.cc:377] Logged in from keytab as oryx/127.10.225.188@KRBTEST.COM (short username oryx)
I20260806 23:52:17.638039 21486 webserver.cc:460] Webserver started at http://127.10.225.188:37915/ using document root <none> and password file <none>
I20260806 23:52:17.638227 21486 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:17.638262 21486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:17.639698 21486 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:17.640777 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.641052 21486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:17.641126 21486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal
uuid: "42b3078ff45f478a98e40232ca8dad04"
format_stamp: "Formatted at 2026-08-06 23:52:17 on dist-test-slave-nj21"
server_key: "9d8a0c3ca7e281c096ccef70e59be30f"
server_key_iv: "d04b85dfb87159478ac1d0708ec9c670"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.641206 21486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:17.681350 21486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:17.683914 21486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:17.689013 21486 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.188:38663
I20260806 23:52:17.689617 21878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.188:38663 every 8 connection(s)
I20260806 23:52:17.689848 21486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
W20260806 23:52:17.689716 21770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:17.690042 21770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:17.690119 21770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 23:52:17.691942 21881 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:52:17.693085 21881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: Bootstrap starting.
W20260806 23:52:17.696702 21770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:17.696786 21770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:17.696879 21770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:17.703586 21770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab
--principal=oryx/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:33373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=35813
--tserver_master_addrs=127.10.225.190:46319,127.10.225.189:39079,127.10.225.188:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:17.704792 21770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:17.705878 21770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:52:17.706851 21881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:52:17.707214 21881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: Bootstrap complete.
I20260806 23:52:17.707917 21881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } }
I20260806 23:52:17.708086 21881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42b3078ff45f478a98e40232ca8dad04, State: Initialized, Role: LEARNER
I20260806 23:52:17.708204 21881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } }
I20260806 23:52:17.708655 21886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:17.708791 21886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: This master's current role is: LEARNER
I20260806 23:52:17.708815 21881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:17.712384 21902 catalog_manager.cc:1259] Loaded cluster ID: 97d8fd91848547a78de6f9001ce17931
I20260806 23:52:17.712461 21902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: loading cluster ID for follower catalog manager: success
I20260806 23:52:17.714728 21902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: acquiring CA information for follower catalog manager: success
I20260806 23:52:17.715997 21902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 23:52:17.719261 21909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:17.719286 21770 server_base.cc:1034] running on GCE node
W20260806 23:52:17.719221 21912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.719310 21907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:17.719992 21770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:17.720712 21770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:17.721891 21770 hybrid_clock.cc:648] HybridClock initialized: now 1786060337721871 us; error 39 us; skew 500 ppm
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:17.725257 21770 init.cc:377] Logged in from keytab as oryx/127.10.225.129@KRBTEST.COM (short username oryx)
I20260806 23:52:17.726354 21770 webserver.cc:460] Webserver started at http://127.10.225.129:35813/ using document root <none> and password file <none>
I20260806 23:52:17.726981 21770 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:17.727075 21770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:17.731747 21770 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:52:17.734813 21922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.736400 21770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:17.736553 21770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "07476883c4614a9c88ef41245bb4b35c"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "c9776a1b5a2c8ccb1982ccdaad35a930"
server_key_iv: "096cb64fa0c84c70d8a7642ffb8c7795"
server_key_version: "encryptionkey@0"
I20260806 23:52:17.737178 21770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:17.755311 21770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:17.759001 21770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:17.759177 21770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:17.760355 21770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:17.761648 21770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:17.761704 21770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.761754 21770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:17.761780 21770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.778399 21770 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:33373
I20260806 23:52:17.778420 22083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:33373 every 8 connection(s)
I20260806 23:52:17.779585 21770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:17.785960 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 21770
I20260806 23:52:17.786226 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 20918
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.188@KRBTEST.COM
I20260806 23:52:17.800390 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.781977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:48341 (local address 127.10.225.190:46319)
0806 23:52:17.782223 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.782227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.783179 (+ 952us) server_negotiation.cc:407] Connection header received
0806 23:52:17.784404 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.784407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.784470 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.784524 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.786977 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.787858 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.789707 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.790001 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.790069 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.794903 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.794910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.794914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.794959 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.797413 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.798277 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.798281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.798282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.798376 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.798822 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.798825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.798826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.799047 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.799127 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.800069 (+ 942us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.800208 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260806 23:52:17.803014 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.782300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.129:48341)
0806 23:52:17.783023 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.783076 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.784135 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.784848 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.784855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.785451 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.786867 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.786875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.788093 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.788096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.789479 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.789492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.790351 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.790355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.790589 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.792264 (+ 1675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.792290 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.794626 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.797611 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.797621 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.797647 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.798147 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.798480 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.798485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.798487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.798728 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.799158 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.799167 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.799661 (+ 494us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.800015 (+ 354us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.800847 (+ 832us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":626,"spinlock_wait_cycles":24064,"thread_start_us":361,"threads_started":1}
I20260806 23:52:17.805763 22086 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46319
I20260806 23:52:17.806095 22086 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:17.806721 22086 heartbeater.cc:511] Master 127.10.225.190:46319 requested a full tablet report, sending...
I20260806 23:52:17.809006 20606 ts_manager.cc:194] Re-registered known tserver with Master: 07476883c4614a9c88ef41245bb4b35c (127.10.225.129:33373)
I20260806 23:52:17.810014 20606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:48341
I20260806 23:52:17.818012 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.781945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:35961 (local address 127.10.225.188:38663)
0806 23:52:17.782248 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.782262 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.784250 (+ 1988us) server_negotiation.cc:407] Connection header received
0806 23:52:17.784362 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.784365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.784436 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.784535 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.787103 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.788571 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.790149 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.790576 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.790669 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.795187 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.795191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.795194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.795225 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.815179 (+ 19954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.816121 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.816127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.816130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.816288 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.816727 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.816731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.816734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.817071 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.817180 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.817637 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.817815 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
I20260806 23:52:17.818039 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.782387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:38663 (local address 127.10.225.129:35961)
0806 23:52:17.784131 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.784142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.784249 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.784844 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.784854 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.785446 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.786863 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.786895 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.788763 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.788768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.789977 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.789993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.790898 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.790903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.791018 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.791959 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.791988 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.795033 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.815379 (+ 20346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.815385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.815388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.815887 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.816450 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.816454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.816457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.816581 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.817251 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.817255 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.817389 (+ 134us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.817529 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.817749 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1687,"mutex_wait_us":165,"spinlock_wait_cycles":2816,"thread_start_us":1037,"threads_started":1}
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.189@KRBTEST.COM
I20260806 23:52:17.823820 21329 heartbeater.cc:503] Master 127.10.225.190:46319 was elected leader, sending a full tablet report...
I20260806 23:52:17.825680 22084 heartbeater.cc:347] Connected to a master server at 127.10.225.188:38663
I20260806 23:52:17.825837 22084 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:17.826262 22084 heartbeater.cc:511] Master 127.10.225.188:38663 requested a full tablet report, sending...
I20260806 23:52:17.828094 21818 ts_manager.cc:194] Registered new tserver with Master: 07476883c4614a9c88ef41245bb4b35c (127.10.225.129:33373)
I20260806 23:52:17.828291 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.781832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:40953 (local address 127.10.225.189:39079)
0806 23:52:17.782124 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:17.782128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:17.783919 (+ 1791us) server_negotiation.cc:407] Connection header received
0806 23:52:17.784492 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:17.784495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:17.784565 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:17.784634 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:17.814291 (+ 29657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.815183 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.821070 (+ 5887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.821325 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.821432 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.824505 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:17.824510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:17.824513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:17.824551 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:17.826167 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.826664 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.826668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.826669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.826869 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:17.827250 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:17.827254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:17.827256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:17.827501 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:17.827599 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:17.827960 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 23:52:17.828113 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260806 23:52:17.828678 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:17.782300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.10.225.129:40953)
0806 23:52:17.783773 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:17.783793 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:17.784145 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:17.784848 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:17.784855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:17.785451 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:17.786339 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.786360 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.820247 (+ 33887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.820252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.820918 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:17.820930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:17.821477 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:17.821481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:17.821585 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:17.822197 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:17.822210 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:17.824280 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:17.826303 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.826306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.826308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.826556 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.827014 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:17.827018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:17.827020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:17.827137 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:17.827627 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:17.827631 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:17.827783 (+ 152us) client_negotiation.cc:769] Sending connection context
0806 23:52:17.827871 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 23:52:17.828105 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1319,"mutex_wait_us":154,"spinlock_wait_cycles":70656,"thread_start_us":220,"threads_started":1}
I20260806 23:52:17.829792 22087 heartbeater.cc:347] Connected to a master server at 127.10.225.189:39079
I20260806 23:52:17.829926 22087 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:17.830122 22087 heartbeater.cc:511] Master 127.10.225.189:39079 requested a full tablet report, sending...
I20260806 23:52:17.831452 21542 ts_manager.cc:194] Registered new tserver with Master: 07476883c4614a9c88ef41245bb4b35c (127.10.225.129:33373)
I20260806 23:52:17.832067 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:34931
--local_ip_for_outbound_sockets=127.10.225.130
--tserver_master_addrs=127.10.225.190:46319
--webserver_port=33171
--webserver_interface=127.10.225.130
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab
--principal=oryx/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.225.190:46319,127.10.225.189:39079
--tserver_master_addrs=127.10.225.190:46319,127.10.225.189:39079,127.10.225.188:38663 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:17.959595 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:17.959910 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:17.959964 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:17.965919 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:17.966006 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:17.966076 22096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:17.972626 22096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab
--principal=oryx/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:34931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=33171
--tserver_master_addrs=127.10.225.190:46319,127.10.225.189:39079,127.10.225.188:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:17.973799 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:17.974884 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:52:17.985044 22096 server_base.cc:1034] running on GCE node
W20260806 23:52:17.984964 22101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.984962 22103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:17.985001 22105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:17.985591 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:17.986224 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:17.987406 22096 hybrid_clock.cc:648] HybridClock initialized: now 1786060337987415 us; error 60 us; skew 500 ppm
Aug 06 23:52:17 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:17.990301 22096 init.cc:377] Logged in from keytab as oryx/127.10.225.130@KRBTEST.COM (short username oryx)
I20260806 23:52:17.991273 22096 webserver.cc:460] Webserver started at http://127.10.225.130:33171/ using document root <none> and password file <none>
I20260806 23:52:17.991904 22096 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:17.991997 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:17.996091 22096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:52:17.998361 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:17.999780 22096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:17.999902 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "f8b80aa9e9c647a1af02c6718b38d4a3"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "b5b7fceccbf8b506fd339261caccf2ba"
server_key_iv: "bc6150840de8f7263f0575a5c02d4045"
server_key_version: "encryptionkey@0"
I20260806 23:52:18.000463 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:18.014706 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:18.018544 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:18.018759 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:18.019416 22096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:18.020465 22096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:18.020517 22096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:18.020583 22096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:18.020609 22096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:18.030555 22096 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:34931
I20260806 23:52:18.030586 22271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:34931 every 8 connection(s)
I20260806 23:52:18.031628 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:18.040208 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 22096
I20260806 23:52:18.040571 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 21136
Aug 06 23:52:18 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.188@KRBTEST.COM
Aug 06 23:52:18 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:18.049500 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.033523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:54411 (local address 127.10.225.190:46319)
0806 23:52:18.033828 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.033833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.035114 (+ 1281us) server_negotiation.cc:407] Connection header received
0806 23:52:18.035701 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.035705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.035759 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.035809 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.037111 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.037888 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.039190 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.039420 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.039512 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.045262 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.045266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.045269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.045300 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.047208 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.047827 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.047831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.047833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.047916 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.048263 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.048266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.048267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.048488 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.048553 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.049192 (+ 639us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.049318 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260806 23:52:18.051041 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.033974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.130:54411)
0806 23:52:18.035029 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.035041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.035469 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.035996 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.036005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.036452 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:18.036934 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.036951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.037997 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.038000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.039050 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.039061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.039649 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.039653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.039791 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.040532 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:18.040556 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:18.045106 (+ 4550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:18.047345 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.047352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.047380 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.047728 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.048012 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.048014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.048016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.048180 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.048629 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:18.048636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:18.048954 (+ 318us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.049683 (+ 729us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.050038 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":280,"spinlock_wait_cycles":67072,"thread_start_us":131,"threads_started":1}
I20260806 23:52:18.062060 22273 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46319
I20260806 23:52:18.062376 22273 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:18.063020 22273 heartbeater.cc:511] Master 127.10.225.190:46319 requested a full tablet report, sending...
I20260806 23:52:18.064492 20606 ts_manager.cc:194] Re-registered known tserver with Master: f8b80aa9e9c647a1af02c6718b38d4a3 (127.10.225.130:34931)
I20260806 23:52:18.065253 20606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:54411
I20260806 23:52:18.065419 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.034131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:59635 (local address 127.10.225.188:38663)
0806 23:52:18.034970 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.034974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.034991 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:52:18.035701 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.035704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.035759 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.035809 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.037090 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.037618 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.039160 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.039492 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.039588 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.056808 (+ 17220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.056813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.056816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.056859 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.061321 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.063759 (+ 2438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.063763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.063765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.063879 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.064200 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.064204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.064205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.064568 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.064655 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.065069 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.065234 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753}
Aug 06 23:52:18 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060337, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.189@KRBTEST.COM
I20260806 23:52:18.066030 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.033974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:38663 (local address 127.10.225.130:59635)
0806 23:52:18.034472 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.034524 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.035458 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.035996 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.036005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.036451 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:18.036934 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.036951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.037756 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.037758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.039051 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.039061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.039603 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.039607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.039732 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.040301 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:18.040324 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:18.044734 (+ 4410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:18.063271 (+ 18537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.063288 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.063291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.063583 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.064001 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.064005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.064007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.064093 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.064738 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:18.064741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:18.064848 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.065659 (+ 811us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.065820 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":143,"spinlock_wait_cycles":33920,"thread_start_us":128,"threads_started":1}
I20260806 23:52:18.066732 22272 heartbeater.cc:347] Connected to a master server at 127.10.225.188:38663
I20260806 23:52:18.066840 22272 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:18.067616 22272 heartbeater.cc:511] Master 127.10.225.188:38663 requested a full tablet report, sending...
I20260806 23:52:18.069581 21818 ts_manager.cc:194] Registered new tserver with Master: f8b80aa9e9c647a1af02c6718b38d4a3 (127.10.225.130:34931)
I20260806 23:52:18.070655 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.033974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.10.225.130:57977)
0806 23:52:18.034990 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.035003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.035461 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.035996 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.036002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.036409 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:18.036934 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.036947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.038033 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.038036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.038906 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.038916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.039477 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.039482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.039693 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.063810 (+ 24117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:18.063824 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:18.066121 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:18.068348 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.068352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.068353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.068822 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.069226 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.069230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.069232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.069358 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.070039 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:18.070043 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:18.070169 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.070329 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.070517 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":117,"thread_start_us":133,"threads_started":1}
I20260806 23:52:18.070734 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.033673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:57977 (local address 127.10.225.189:39079)
0806 23:52:18.033921 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.033926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.035082 (+ 1156us) server_negotiation.cc:407] Connection header received
0806 23:52:18.035653 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.035656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.035737 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.035822 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.037064 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.037888 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.039033 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.039316 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.039424 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.066318 (+ 26894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.066323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.066326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.066370 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.068226 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.068978 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.068981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.068983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.069081 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.069537 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.069543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.069545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.069862 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.070039 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.070424 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.070569 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260806 23:52:18.071302 22274 heartbeater.cc:347] Connected to a master server at 127.10.225.189:39079
I20260806 23:52:18.071408 22274 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:18.071617 22274 heartbeater.cc:511] Master 127.10.225.189:39079 requested a full tablet report, sending...
I20260806 23:52:18.071957 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:44527
--local_ip_for_outbound_sockets=127.10.225.131
--tserver_master_addrs=127.10.225.190:46319
--webserver_port=39789
--webserver_interface=127.10.225.131
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab
--principal=oryx/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.225.190:46319,127.10.225.189:39079
--tserver_master_addrs=127.10.225.190:46319,127.10.225.189:39079,127.10.225.188:38663 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:52:18.072412 21542 ts_manager.cc:194] Registered new tserver with Master: f8b80aa9e9c647a1af02c6718b38d4a3 (127.10.225.130:34931)
I20260806 23:52:18.144119 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.139904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:38663 (local address 127.0.0.1:34002)
0806 23:52:18.140252 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.140270 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.140418 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.140682 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.140685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.140705 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:18.140815 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.140822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.141646 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.141650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.142902 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.142913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.143651 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.143654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.143770 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.143796 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.143885 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.143980 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":124,"threads_started":1}
I20260806 23:52:18.144117 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.139989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.10.225.188:38663)
0806 23:52:18.140124 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.140130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.140370 (+ 240us) server_negotiation.cc:407] Connection header received
0806 23:52:18.140523 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.140527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.140577 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.140659 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:18.140926 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.141537 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.143023 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.143531 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.143650 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.143819 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.143890 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.143963 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:52:18.145402 21839 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 23:52:18.145885 21886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8398b82fa36145bba7638367222ecd03. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:18.146111 21886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: This master's current role is: LEARNER
W20260806 23:52:18.196388 22282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:18.196826 22282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:18.196899 22282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:18.201558 22282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:18.201632 22282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:18.201722 22282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:18.207391 22282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab
--principal=oryx/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:44527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=39789
--tserver_master_addrs=127.10.225.190:46319,127.10.225.189:39079,127.10.225.188:38663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:18.208665 22282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:18.209838 22282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:18.221204 22293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:18.221207 22289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:18.221227 22291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:18.221500 22282 server_base.cc:1034] running on GCE node
I20260806 23:52:18.222072 22282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:18.222831 22282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:18.224016 22282 hybrid_clock.cc:648] HybridClock initialized: now 1786060338223987 us; error 47 us; skew 500 ppm
Aug 06 23:52:18 dist-test-slave-nj21 krb5kdc[20540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060338, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:18.227191 22282 init.cc:377] Logged in from keytab as oryx/127.10.225.131@KRBTEST.COM (short username oryx)
I20260806 23:52:18.228174 22282 webserver.cc:460] Webserver started at http://127.10.225.131:39789/ using document root <none> and password file <none>
I20260806 23:52:18.228734 22282 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:18.228809 22282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:18.232963 22282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 23:52:18.235168 22303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:18.236791 22282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:52:18.236925 22282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "b9bf272e79cc447db72a89c8e52aa8f9"
format_stamp: "Formatted at 2026-08-06 23:52:16 on dist-test-slave-nj21"
server_key: "86aa4cd650d315dd7b3a1c75f4987da1"
server_key_iv: "aa2728c433fbc656732ac2ab8aa20476"
server_key_version: "encryptionkey@0"
I20260806 23:52:18.237540 22282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:18.253849 22282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:18.257210 22282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:18.257416 22282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:18.258052 22282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:18.258965 22282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:18.259012 22282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:18.259069 22282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:18.259096 22282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:18.268832 22282 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:44527
I20260806 23:52:18.268878 22460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:44527 every 8 connection(s)
I20260806 23:52:18.270022 22282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:18.270545 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 22282
Aug 06 23:52:18 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060338, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.188@KRBTEST.COM
I20260806 23:52:18.285917 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.274088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.10.225.188:38663)
0806 23:52:18.274355 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.274359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.274371 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:52:18.274426 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.274429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.274478 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.274560 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.275908 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.276752 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.277967 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.278328 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.278403 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.281730 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.281735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.281738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.281774 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.283793 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.284462 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.284466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.284468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.284564 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.284862 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.284865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.284867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.285098 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.285198 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.285564 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.285710 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":217,"thread_start_us":67,"threads_started":1}
Aug 06 23:52:18 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060338, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
Aug 06 23:52:18 dist-test-slave-nj21 krb5kdc[20540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060338, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.189@KRBTEST.COM
I20260806 23:52:18.288705 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.273994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:48491 (local address 127.10.225.188:38663)
0806 23:52:18.274267 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.274271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.274544 (+ 273us) server_negotiation.cc:407] Connection header received
0806 23:52:18.275796 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.275800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.275855 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.275904 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.277926 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.278713 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.280466 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.280774 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.280842 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.284257 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.284262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.284265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.284290 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.285940 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.286555 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.286559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.286560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.286647 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.287026 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.287029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.287031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.287270 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.287351 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.288414 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.288539 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":204}
I20260806 23:52:18.290735 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.272733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:38663 (local address 127.10.225.131:48491)
0806 23:52:18.274393 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.274453 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.275545 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.276349 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.276369 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.276964 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:18.277766 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.277786 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.278891 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.278894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.280328 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.280339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.281039 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.281043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.281251 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.281903 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:18.281924 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:18.284069 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:18.286063 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.286069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.286089 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.286451 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.286748 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.286751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.286753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.286927 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.287702 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:18.287709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:18.288080 (+ 371us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.288861 (+ 781us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.289239 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"mutex_wait_us":63,"spinlock_wait_cycles":27776,"thread_start_us":672,"threads_started":1}
I20260806 23:52:18.291795 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.272390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:38561 (local address 127.10.225.189:39079)
0806 23:52:18.272548 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.272553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.274787 (+ 2234us) server_negotiation.cc:407] Connection header received
0806 23:52:18.275847 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.275850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.275899 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.275943 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.279781 (+ 3838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.280503 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.281824 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.283035 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.283114 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.288030 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.288034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.288037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.288064 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.289606 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.290225 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.290230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.290232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.290323 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.290679 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.290683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.290685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.290927 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.291018 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.291478 (+ 460us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.291605 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 23:52:18.292135 22461 heartbeater.cc:347] Connected to a master server at 127.10.225.188:38663
I20260806 23:52:18.292367 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.273129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:47641 (local address 127.10.225.190:46319)
0806 23:52:18.273255 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.273260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.274543 (+ 1283us) server_negotiation.cc:407] Connection header received
0806 23:52:18.276654 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.276657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.276717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.276775 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.277940 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.278713 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.282509 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.283253 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.283349 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.287053 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.287057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.287060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.287092 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.289281 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.290230 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.290234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.290236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.290326 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.290645 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.290648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.290650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.290884 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.290969 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.291742 (+ 773us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.291874 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:52:18.292423 22461 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:18.293011 22461 heartbeater.cc:511] Master 127.10.225.188:38663 requested a full tablet report, sending...
I20260806 23:52:18.293232 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.273015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.131:47641)
0806 23:52:18.274393 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.274453 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.275580 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.276940 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.276943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.277107 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:18.277766 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.277786 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.279167 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.279171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.282349 (+ 3178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.282363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.284103 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.284107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.284205 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.284888 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:18.284903 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:18.286897 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:18.289415 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.289417 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.289419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.290115 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.290429 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.290432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.290434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.290534 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.291444 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:18.291447 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:18.291530 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.292893 (+ 1363us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.293041 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":2468,"spinlock_wait_cycles":2944,"thread_start_us":110,"threads_started":1}
I20260806 23:52:18.293895 22462 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46319
I20260806 23:52:18.293992 22462 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:18.294193 22462 heartbeater.cc:511] Master 127.10.225.190:46319 requested a full tablet report, sending...
I20260806 23:52:18.294365 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.273991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.10.225.131:38561)
0806 23:52:18.274711 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.274723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.275750 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.276349 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.276358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.277316 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:18.279613 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.279624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.280625 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.280628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.281411 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.281421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.283613 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.283616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.283736 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.285388 (+ 1652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:18.285403 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:18.287855 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:18.289727 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.289731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.289732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.290105 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.290431 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:18.290434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:18.290436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:18.290572 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:18.291184 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:18.291186 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:18.291286 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.294025 (+ 2739us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.294193 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":1901,"spinlock_wait_cycles":5120,"thread_start_us":105,"threads_started":1}
I20260806 23:52:18.294353 21818 ts_manager.cc:194] Registered new tserver with Master: b9bf272e79cc447db72a89c8e52aa8f9 (127.10.225.131:44527)
I20260806 23:52:18.294806 20606 ts_manager.cc:194] Re-registered known tserver with Master: b9bf272e79cc447db72a89c8e52aa8f9 (127.10.225.131:44527)
I20260806 23:52:18.295526 20606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:47641
I20260806 23:52:18.296094 22464 heartbeater.cc:347] Connected to a master server at 127.10.225.189:39079
I20260806 23:52:18.296183 22464 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:18.296350 22464 heartbeater.cc:511] Master 127.10.225.189:39079 requested a full tablet report, sending...
I20260806 23:52:18.296681 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.273979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.10.225.189:39079)
0806 23:52:18.274843 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.274847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.275319 (+ 472us) server_negotiation.cc:407] Connection header received
0806 23:52:18.275365 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.275368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.275414 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.275491 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.281230 (+ 5739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.283188 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.285320 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.285664 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.285755 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.291664 (+ 5909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.291700 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.291703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.291727 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.294320 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.295042 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.295045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.295047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.295145 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.295455 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.295459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.295460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.295700 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.295832 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.296330 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.296525 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":67,"threads_started":1}
I20260806 23:52:18.296880 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.273475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.10.225.190:46319)
0806 23:52:18.280258 (+ 6783us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.280263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.280281 (+ 18us) server_negotiation.cc:407] Connection header received
0806 23:52:18.280334 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.280337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.280397 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.280500 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.284462 (+ 3962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.285200 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.287141 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.287517 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.287614 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.292061 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.292065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.292067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.292094 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.293906 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.295447 (+ 1541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.295451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.295452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.295531 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.295849 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.295852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.295854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.296065 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.296181 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.296596 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.296725 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6678,"thread_start_us":73,"threads_started":1}
I20260806 23:52:18.297931 21542 ts_manager.cc:194] Registered new tserver with Master: b9bf272e79cc447db72a89c8e52aa8f9 (127.10.225.131:44527)
I20260806 23:52:18.300923 20606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:52:18.304229 20606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:18.307088 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: NON_VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: true } } }
I20260806 23:52:18.307233 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:18.311110 22486 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: attempting to promote NON_VOTER 42b3078ff45f478a98e40232ca8dad04 to VOTER
I20260806 23:52:18.311436 22486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } }
I20260806 23:52:18.312748 21559 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Refusing update from remote peer 8398b82fa36145bba7638367222ecd03: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 23:52:18.313285 22489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 23:52:18.314100 21839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 LEARNER]: Refusing update from remote peer 8398b82fa36145bba7638367222ecd03: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 23:52:18.314559 22489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 23:52:18.315654 22489 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:18.316098 21559 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:18.317404 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:18.317520 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:18.317663 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:18.317747 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:18.317585 22489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:18.317806 22489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:18.318012 22502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:18.318132 22502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:52:18.317737 21839 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:18.320727 21886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8398b82fa36145bba7638367222ecd03" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:18.320878 21886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:52:18.326751 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.320825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:44527 (local address 127.0.0.1:59606)
0806 23:52:18.320949 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.320967 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.321052 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.322518 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.322521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.322538 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:18.322650 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.322656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.323959 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.323962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.325208 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.325217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.326340 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.326343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.326434 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.326472 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.326522 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.326604 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 23:52:18.327608 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.321258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.10.225.131:44527)
0806 23:52:18.322020 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.322032 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.322058 (+ 26us) server_negotiation.cc:407] Connection header received
0806 23:52:18.322149 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.322155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.322390 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.322540 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:18.322736 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.323818 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.325351 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.326207 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.327132 (+ 925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.327293 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.327351 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.327458 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":130,"threads_started":1}
I20260806 23:52:18.327780 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.321572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.10.225.129:33373)
0806 23:52:18.322132 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.322139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.322164 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:52:18.323037 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.323043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.323228 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.323389 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:18.323807 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.324896 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.326344 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.327122 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.327236 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.327335 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.327522 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.327617 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":66,"threads_started":1}
I20260806 23:52:18.328401 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.321396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:33373 (local address 127.0.0.1:35876)
0806 23:52:18.322079 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.322093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.322176 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.323541 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.323544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.323564 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:18.323704 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.323712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.325028 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.325032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.326191 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.326203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.327243 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.327248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.327359 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.327392 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.327544 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.327655 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":125,"threads_started":1}
I20260806 23:52:18.329540 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.321849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:34931 (local address 127.0.0.1:56242)
0806 23:52:18.323691 (+ 1842us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.323705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.323788 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.325363 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.325366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.325390 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:18.325557 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.325562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.326722 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.326726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.327966 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.327977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.329090 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.329094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.329190 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.329219 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.329304 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.329389 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1756,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260806 23:52:18.329581 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.322323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.10.225.130:34931)
0806 23:52:18.324234 (+ 1911us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.324254 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.324280 (+ 26us) server_negotiation.cc:407] Connection header received
0806 23:52:18.324629 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.324635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.325000 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.325234 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:18.325783 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.326591 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.328137 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.328963 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.329094 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.329222 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.329318 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.329426 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1725,"thread_start_us":118,"threads_started":1}
I20260806 23:52:18.330992 22366 tablet_service.cc:1467] Processing CreateTablet for tablet 0a610aef54354df89a258c75ca006126 (DEFAULT_TABLE table=test-table [id=6b831b3b04374be49610c29f1233c079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:18.331243 21991 tablet_service.cc:1467] Processing CreateTablet for tablet 0a610aef54354df89a258c75ca006126 (DEFAULT_TABLE table=test-table [id=6b831b3b04374be49610c29f1233c079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:18.332219 21991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a610aef54354df89a258c75ca006126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:18.332237 22366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a610aef54354df89a258c75ca006126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:18.332332 22180 tablet_service.cc:1467] Processing CreateTablet for tablet 0a610aef54354df89a258c75ca006126 (DEFAULT_TABLE table=test-table [id=6b831b3b04374be49610c29f1233c079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:18.333299 22180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a610aef54354df89a258c75ca006126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:18.339354 22508 tablet_bootstrap.cc:492] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: Bootstrap starting.
I20260806 23:52:18.340883 22509 tablet_bootstrap.cc:492] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: Bootstrap starting.
I20260806 23:52:18.342003 22510 tablet_bootstrap.cc:492] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: Bootstrap starting.
I20260806 23:52:18.342348 22508 tablet_bootstrap.cc:654] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:18.342980 22509 tablet_bootstrap.cc:654] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:18.343149 22508 log.cc:824] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:18.343784 22509 log.cc:824] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:18.344265 22510 tablet_bootstrap.cc:654] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:18.344940 22510 log.cc:824] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:18.345161 22508 tablet_bootstrap.cc:492] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: No bootstrap required, opened a new log
I20260806 23:52:18.345567 22508 ts_tablet_manager.cc:1397] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:52:18.345980 22509 tablet_bootstrap.cc:492] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: No bootstrap required, opened a new log
I20260806 23:52:18.346174 22509 ts_tablet_manager.cc:1397] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 23:52:18.347718 22510 tablet_bootstrap.cc:492] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: No bootstrap required, opened a new log
I20260806 23:52:18.347914 22510 ts_tablet_manager.cc:1397] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:52:18.349718 22509 raft_consensus.cc:348] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:18.349730 22508 raft_consensus.cc:348] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:18.349954 22509 raft_consensus.cc:374] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:18.350004 22509 raft_consensus.cc:729] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bf272e79cc447db72a89c8e52aa8f9, State: Initialized, Role: FOLLOWER
I20260806 23:52:18.350013 22508 raft_consensus.cc:374] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:18.350075 22508 raft_consensus.cc:729] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07476883c4614a9c88ef41245bb4b35c, State: Initialized, Role: FOLLOWER
I20260806 23:52:18.350445 22509 consensus_queue.cc:260] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:18.350507 22508 consensus_queue.cc:260] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:18.351482 22509 ts_tablet_manager.cc:1428] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260806 23:52:18.351485 22508 ts_tablet_manager.cc:1428] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:52:18.351758 22462 heartbeater.cc:503] Master 127.10.225.190:46319 was elected leader, sending a full tablet report...
I20260806 23:52:18.352679 22086 heartbeater.cc:503] Master 127.10.225.190:46319 was elected leader, sending a full tablet report...
I20260806 23:52:18.355521 22510 raft_consensus.cc:348] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:18.355866 22510 raft_consensus.cc:374] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:18.355930 22510 raft_consensus.cc:729] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b80aa9e9c647a1af02c6718b38d4a3, State: Initialized, Role: FOLLOWER
I20260806 23:52:18.356462 22510 consensus_queue.cc:260] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:18.357820 22273 heartbeater.cc:503] Master 127.10.225.190:46319 was elected leader, sending a full tablet report...
I20260806 23:52:18.358297 22510 ts_tablet_manager.cc:1428] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260806 23:52:18.409857 22517 raft_consensus.cc:482] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:18.410148 22517 raft_consensus.cc:504] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:18.411427 22517 leader_election.cc:288] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9bf272e79cc447db72a89c8e52aa8f9 (127.10.225.131:44527), f8b80aa9e9c647a1af02c6718b38d4a3 (127.10.225.130:34931)
I20260806 23:52:18.416116 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.411769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:43407 (local address 127.10.225.131:44527)
0806 23:52:18.411919 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.411924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.412171 (+ 247us) server_negotiation.cc:407] Connection header received
0806 23:52:18.412340 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.412343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.412386 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.412449 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:18.412842 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.413703 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.415016 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.415477 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.415613 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.415703 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.415924 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.416010 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 23:52:18.416101 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.411744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:34931 (local address 127.10.225.129:41577)
0806 23:52:18.412072 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.412085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.412237 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.412479 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.412482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.412584 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:18.412725 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.412732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.413779 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.413783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.414729 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.414739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.415584 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.415588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.415717 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.415745 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.415825 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.415908 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":19,"spinlock_wait_cycles":246400,"thread_start_us":138,"threads_started":1}
I20260806 23:52:18.416218 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.411841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:41577 (local address 127.10.225.130:34931)
0806 23:52:18.412036 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.412040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.412156 (+ 116us) server_negotiation.cc:407] Connection header received
0806 23:52:18.412339 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.412342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.412390 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.412453 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:18.412842 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.413646 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.414900 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.415424 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.415553 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.415633 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.415863 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.416037 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 23:52:18.416287 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.411651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:44527 (local address 127.10.225.129:43407)
0806 23:52:18.412063 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:18.412076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:18.412230 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:18.412470 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:18.412473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:18.412570 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:18.412724 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.412732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.413842 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.413847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.414821 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:18.414839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.415637 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.415641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:18.415777 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.415799 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:52:18.415886 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 23:52:18.416037 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":1,"spinlock_wait_cycles":15104,"thread_start_us":133,"threads_started":1}
I20260806 23:52:18.417025 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a610aef54354df89a258c75ca006126" candidate_uuid: "07476883c4614a9c88ef41245bb4b35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" is_pre_election: true
I20260806 23:52:18.417375 22400 raft_consensus.cc:2454] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07476883c4614a9c88ef41245bb4b35c in term 0.
I20260806 23:52:18.417408 22204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a610aef54354df89a258c75ca006126" candidate_uuid: "07476883c4614a9c88ef41245bb4b35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" is_pre_election: true
I20260806 23:52:18.417708 22204 raft_consensus.cc:2454] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07476883c4614a9c88ef41245bb4b35c in term 0.
I20260806 23:52:18.417857 21927 leader_election.cc:302] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07476883c4614a9c88ef41245bb4b35c, b9bf272e79cc447db72a89c8e52aa8f9; no voters:
I20260806 23:52:18.418150 22517 raft_consensus.cc:2781] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:18.418229 22517 raft_consensus.cc:482] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:18.418259 22517 raft_consensus.cc:3037] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:18.419395 22517 raft_consensus.cc:504] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:18.419806 22517 leader_election.cc:288] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [CANDIDATE]: Term 1 election: Requested vote from peers b9bf272e79cc447db72a89c8e52aa8f9 (127.10.225.131:44527), f8b80aa9e9c647a1af02c6718b38d4a3 (127.10.225.130:34931)
I20260806 23:52:18.420209 22204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a610aef54354df89a258c75ca006126" candidate_uuid: "07476883c4614a9c88ef41245bb4b35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3"
I20260806 23:52:18.420353 22204 raft_consensus.cc:3037] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:18.420544 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a610aef54354df89a258c75ca006126" candidate_uuid: "07476883c4614a9c88ef41245bb4b35c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bf272e79cc447db72a89c8e52aa8f9"
I20260806 23:52:18.420764 22400 raft_consensus.cc:3037] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:18.421494 22204 raft_consensus.cc:2454] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07476883c4614a9c88ef41245bb4b35c in term 1.
I20260806 23:52:18.421804 21928 leader_election.cc:302] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07476883c4614a9c88ef41245bb4b35c, f8b80aa9e9c647a1af02c6718b38d4a3; no voters:
I20260806 23:52:18.421978 22517 raft_consensus.cc:2781] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:18.422204 22517 raft_consensus.cc:686] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 1 LEADER]: Becoming Leader. State: Replica: 07476883c4614a9c88ef41245bb4b35c, State: Running, Role: LEADER
I20260806 23:52:18.422582 22517 consensus_queue.cc:237] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:18.422356 22400 raft_consensus.cc:2454] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07476883c4614a9c88ef41245bb4b35c in term 1.
I20260806 23:52:18.427134 20606 catalog_manager.cc:5495] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c reported cstate change: term changed from 0 to 1, leader changed from <none> to 07476883c4614a9c88ef41245bb4b35c (127.10.225.129). New cstate: current_term: 1 leader_uuid: "07476883c4614a9c88ef41245bb4b35c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } health_report { overall_health: HEALTHY } } }
I20260806 23:52:18.460294 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.456820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.10.225.129:33373)
0806 23:52:18.456940 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.456944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.456957 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:52:18.457041 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.457043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.457098 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.457190 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:18.457428 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.457934 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.458767 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.458979 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.459064 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.459375 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:18.459915 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:18.460031 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.460085 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.460147 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:52:18.466645 22400 raft_consensus.cc:1261] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [term 1 FOLLOWER]: Refusing update from remote peer 07476883c4614a9c88ef41245bb4b35c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:18.466645 22204 raft_consensus.cc:1261] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 1 FOLLOWER]: Refusing update from remote peer 07476883c4614a9c88ef41245bb4b35c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:18.467336 22525 consensus_queue.cc:1035] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:18.467609 22517 consensus_queue.cc:1035] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:18.478535 22532 mvcc.cc:205] Tried to move back new op lower bound from 7315703146351915008 to 7315703146182184960. Current Snapshot: MvccSnapshot[applied={T|T < 7315703146351915008}]
I20260806 23:52:18.478593 22530 mvcc.cc:205] Tried to move back new op lower bound from 7315703146351915008 to 7315703146182184960. Current Snapshot: MvccSnapshot[applied={T|T < 7315703146351915008}]
I20260806 23:52:18.488914 20606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:18.489095 20606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:18.493839 20606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8398b82fa36145bba7638367222ecd03: Sending DeleteTablet for 3 replicas of tablet 0a610aef54354df89a258c75ca006126
I20260806 23:52:18.494663 22366 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a610aef54354df89a258c75ca006126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:18 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:59606
I20260806 23:52:18.494755 22180 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a610aef54354df89a258c75ca006126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:18 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:56242
I20260806 23:52:18.494864 21991 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a610aef54354df89a258c75ca006126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:18 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:35876
I20260806 23:52:18.495364 22544 tablet_replica.cc:330] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: stopping tablet replica
I20260806 23:52:18.495764 22544 raft_consensus.cc:2229] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:52:18.496232 22544 raft_consensus.cc:2258] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:18.496640 22541 tablet_replica.cc:330] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: stopping tablet replica
I20260806 23:52:18.496945 22541 raft_consensus.cc:2229] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:18.497244 22545 tablet_replica.cc:330] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: stopping tablet replica
I20260806 23:52:18.497277 22541 raft_consensus.cc:2258] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:18.497551 22545 raft_consensus.cc:2229] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:18.497866 22545 raft_consensus.cc:2258] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:18.498009 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 20559
I20260806 23:52:18.508394 22544 ts_tablet_manager.cc:1905] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:18.510066 22545 ts_tablet_manager.cc:1905] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:18.510726 22541 ts_tablet_manager.cc:1905] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:18.512887 22545 ts_tablet_manager.cc:1918] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:18.513016 22545 log.cc:1196] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/0a610aef54354df89a258c75ca006126
I20260806 23:52:18.513389 22545 ts_tablet_manager.cc:1939] T 0a610aef54354df89a258c75ca006126 P f8b80aa9e9c647a1af02c6718b38d4a3: Deleting consensus metadata
I20260806 23:52:18.514071 22544 ts_tablet_manager.cc:1918] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:18.514183 22544 log.cc:1196] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/0a610aef54354df89a258c75ca006126
I20260806 23:52:18.514503 22544 ts_tablet_manager.cc:1939] T 0a610aef54354df89a258c75ca006126 P 07476883c4614a9c88ef41245bb4b35c: Deleting consensus metadata
I20260806 23:52:18.517045 22541 ts_tablet_manager.cc:1918] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:18.517148 22541 log.cc:1196] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/0a610aef54354df89a258c75ca006126
I20260806 23:52:18.517753 22541 ts_tablet_manager.cc:1939] T 0a610aef54354df89a258c75ca006126 P b9bf272e79cc447db72a89c8e52aa8f9: Deleting consensus metadata
W20260806 23:52:18.518309 21928 connection.cc:535] server connection from 127.0.0.1:35876 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 23:52:18.518361 22113 connection.cc:535] server connection from 127.0.0.1:56242 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 23:52:18.519402 22304 connection.cc:441] server connection from 127.0.0.1:59606 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59606 (request call id 1) could send its response
I20260806 23:52:18.528254 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.496454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.10.225.189:39079)
0806 23:52:18.496810 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.496815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.510498 (+ 13683us) server_negotiation.cc:407] Connection header received
0806 23:52:18.510745 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.510748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.510822 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.510878 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.519920 (+ 9042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.520598 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.521693 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.522085 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.522254 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.524434 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.524439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.524442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.524480 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.526226 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.526754 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.526758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.526760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.526848 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.527143 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.527146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.527148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.527369 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.527483 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.527926 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.528075 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260806 23:52:18.533437 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.522798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.10.225.189:39079)
0806 23:52:18.522938 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.522942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.523736 (+ 794us) server_negotiation.cc:407] Connection header received
0806 23:52:18.523945 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.523948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.524020 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.524075 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.525600 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.526345 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.527468 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.527888 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.527967 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.529906 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.529910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.529913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.529940 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.531499 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.532093 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.532097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.532099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.532185 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.532494 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.532498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.532500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.532732 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.532863 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.533127 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.533254 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:52:18.534144 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.523725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.10.225.188:38663)
0806 23:52:18.523898 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.523903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.523924 (+ 21us) server_negotiation.cc:407] Connection header received
0806 23:52:18.523983 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.523985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.524038 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.524140 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.525349 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.526178 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.527451 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.527917 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.527983 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.529710 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.529714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.529717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.529746 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.531581 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.532093 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.532097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.532099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.532185 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.533190 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.533193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.533195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.533404 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.533502 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.533848 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.533987 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 23:52:18.537644 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:18.496986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.10.225.188:38663)
0806 23:52:18.497105 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:18.497109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:18.529415 (+ 32306us) server_negotiation.cc:407] Connection header received
0806 23:52:18.529630 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:18.529633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:18.529689 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:18.529739 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:18.530493 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.531278 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.532405 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:18.532711 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:18.532783 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:18.534743 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:18.534746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:18.534748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:18.534769 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:18.536135 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.536639 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.536642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.536643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.536722 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:18.536979 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:18.536981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:18.536982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:18.537136 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:18.537233 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:18.537427 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 23:52:18.537515 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:52:19.481563 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.481100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.131:58827)
0806 23:52:19.481364 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:19.481462 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260806 23:52:19.483489 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.482773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.130:47321)
0806 23:52:19.483287 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:19.483379 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":66,"threads_started":1}
I20260806 23:52:19.483743 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.482672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.129:50859)
0806 23:52:19.483453 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:19.483619 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":70,"threads_started":1}
W20260806 23:52:19.483799 22462 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:46319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
W20260806 23:52:19.483820 22273 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:46319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
W20260806 23:52:19.484030 22086 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:46319 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
I20260806 23:52:19.993928 22561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8398b82fa36145bba7638367222ecd03)
I20260806 23:52:19.994155 22561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } }
I20260806 23:52:19.994495 22565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8398b82fa36145bba7638367222ecd03)
I20260806 23:52:19.994661 22565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } }
I20260806 23:52:19.995263 22561 leader_election.cc:288] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319), 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188:38663)
I20260806 23:52:19.995522 22565 leader_election.cc:288] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319), 967fd0ceedd74c40aa33e18b1723df9a (127.10.225.189:39079)
I20260806 23:52:19.996030 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.995489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53608)
0806 23:52:19.995835 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:19.995920 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":69,"threads_started":1}
W20260806 23:52:19.996207 21518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
I20260806 23:52:19.996342 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.995824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53618)
0806 23:52:19.996180 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:19.996252 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":82,"threads_started":1}
W20260806 23:52:19.996512 21800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
I20260806 23:52:19.997088 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.996836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53620)
0806 23:52:19.996973 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:19.997025 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 23:52:19.997193 21518 leader_election.cc:334] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319): Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
I20260806 23:52:19.997658 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.997087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53622)
0806 23:52:19.997541 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:19.997593 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393}
W20260806 23:52:19.997768 21800 leader_election.cc:334] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319): Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
I20260806 23:52:20.000643 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.995894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:39079 (local address 127.0.0.1:51628)
0806 23:52:19.996715 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:19.996729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:19.996866 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:19.997239 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:19.997242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:19.997259 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:19.997369 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:19.997377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:19.998310 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:19.998314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:19.999448 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:19.999459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:20.000311 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:20.000313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:20.000384 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:20.000414 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 23:52:20.000474 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 23:52:20.000536 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":78,"threads_started":1}
I20260806 23:52:20.000686 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.995943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.10.225.189:39079)
0806 23:52:19.996565 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:19.996570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:19.996795 (+ 225us) server_negotiation.cc:407] Connection header received
0806 23:52:19.996988 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:19.996991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:19.997039 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:19.997092 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:19.997473 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:19.998201 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:19.999614 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:20.000194 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:20.000328 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:20.000383 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:20.000500 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 23:52:20.000566 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":557,"thread_start_us":58,"threads_started":1}
I20260806 23:52:20.001116 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.995504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:38663 (local address 127.0.0.1:34030)
0806 23:52:19.996236 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:19.996249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:19.996371 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:19.997396 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:19.997404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:19.997458 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:19.997681 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:19.997690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:19.998820 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:19.998824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:19.999898 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:19.999908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:20.000798 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:20.000802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:20.000887 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:20.000909 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:52:20.000958 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 23:52:20.001026 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":11,"thread_start_us":62,"threads_started":1}
I20260806 23:52:20.001111 21559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42b3078ff45f478a98e40232ca8dad04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "967fd0ceedd74c40aa33e18b1723df9a" is_pre_election: true
I20260806 23:52:20.001349 21559 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b3078ff45f478a98e40232ca8dad04 in term 1.
I20260806 23:52:20.001483 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:19.996044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.10.225.188:38663)
0806 23:52:19.997009 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:19.997013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:19.997024 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:52:19.997059 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:19.997063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:19.997099 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:19.997312 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:19.997924 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:19.998678 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:20.000029 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:20.000655 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:20.001196 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:20.001257 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:20.001294 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 23:52:20.001370 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":67,"threads_started":1}
I20260806 23:52:20.001704 21802 leader_election.cc:302] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42b3078ff45f478a98e40232ca8dad04, 967fd0ceedd74c40aa33e18b1723df9a; no voters: 8398b82fa36145bba7638367222ecd03
I20260806 23:52:20.001854 21839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "967fd0ceedd74c40aa33e18b1723df9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42b3078ff45f478a98e40232ca8dad04" is_pre_election: true
I20260806 23:52:20.001935 22565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 23:52:20.002058 21839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967fd0ceedd74c40aa33e18b1723df9a in term 1.
I20260806 23:52:20.002133 22565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8398b82fa36145bba7638367222ecd03)
I20260806 23:52:20.002197 22565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:52:20.002398 21521 leader_election.cc:302] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42b3078ff45f478a98e40232ca8dad04, 967fd0ceedd74c40aa33e18b1723df9a; no voters: 8398b82fa36145bba7638367222ecd03
I20260806 23:52:20.002580 22561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 23:52:20.002641 22561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8398b82fa36145bba7638367222ecd03)
I20260806 23:52:20.002681 22561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:52:20.003921 22561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } }
I20260806 23:52:20.004132 22565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } }
I20260806 23:52:20.004300 22561 leader_election.cc:288] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 2 election: Requested vote from peers 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319), 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188:38663)
I20260806 23:52:20.004699 21839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "967fd0ceedd74c40aa33e18b1723df9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42b3078ff45f478a98e40232ca8dad04"
I20260806 23:52:20.004882 21839 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 967fd0ceedd74c40aa33e18b1723df9a in current term 2: Already voted for candidate 42b3078ff45f478a98e40232ca8dad04 in this term.
I20260806 23:52:20.005067 21559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42b3078ff45f478a98e40232ca8dad04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "967fd0ceedd74c40aa33e18b1723df9a"
I20260806 23:52:20.005081 22565 leader_election.cc:288] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [CANDIDATE]: Term 2 election: Requested vote from peers 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319), 967fd0ceedd74c40aa33e18b1723df9a (127.10.225.189:39079)
I20260806 23:52:20.005225 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:20.004578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53630)
0806 23:52:20.005105 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:20.005165 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452}
I20260806 23:52:20.005228 21559 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42b3078ff45f478a98e40232ca8dad04 in current term 2: Already voted for candidate 967fd0ceedd74c40aa33e18b1723df9a in this term.
I20260806 23:52:20.005412 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:20.004828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53634)
0806 23:52:20.005114 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:20.005318 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 23:52:20.006039 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:20.005790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53636)
0806 23:52:20.005920 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:20.005973 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 23:52:20.006145 21518 leader_election.cc:334] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319): Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
I20260806 23:52:20.006235 21518 leader_election.cc:302] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 967fd0ceedd74c40aa33e18b1723df9a; no voters: 42b3078ff45f478a98e40232ca8dad04, 8398b82fa36145bba7638367222ecd03
I20260806 23:52:20.006381 22561 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 23:52:20.006625 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:20.006365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53642)
0806 23:52:20.006511 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:20.006562 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 23:52:20.006735 21800 leader_election.cc:334] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319): Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
I20260806 23:52:20.006839 21800 leader_election.cc:302] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42b3078ff45f478a98e40232ca8dad04; no voters: 8398b82fa36145bba7638367222ecd03, 967fd0ceedd74c40aa33e18b1723df9a
I20260806 23:52:20.006989 22565 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 23:52:20.485261 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:20.484360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.130:33987)
0806 23:52:20.485064 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:20.485139 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":111,"threads_started":1}
I20260806 23:52:20.485802 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:20.484685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.131:57321)
0806 23:52:20.485639 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:20.485707 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":111,"threads_started":1}
I20260806 23:52:20.486028 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:20.484672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.129:49439)
0806 23:52:20.485817 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:20.485892 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1057,"thread_start_us":123,"threads_started":1}
I20260806 23:52:21.486639 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.485970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.130:32907)
0806 23:52:21.486444 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.486522 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":104,"threads_started":1}
I20260806 23:52:21.487175 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.486595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.129:44635)
0806 23:52:21.487036 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.487085 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":63,"threads_started":1}
I20260806 23:52:21.487226 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.486595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.131:57201)
0806 23:52:21.487069 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.487130 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":67,"threads_started":1}
I20260806 23:52:21.712154 22584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:21.712316 22584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } }
I20260806 23:52:21.712772 22584 leader_election.cc:288] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319), 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188:38663)
I20260806 23:52:21.713264 21839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "967fd0ceedd74c40aa33e18b1723df9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42b3078ff45f478a98e40232ca8dad04" is_pre_election: true
I20260806 23:52:21.713372 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.712994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53926)
0806 23:52:21.713201 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.713275 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":55,"threads_started":1}
I20260806 23:52:21.713413 21839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 967fd0ceedd74c40aa33e18b1723df9a in term 2.
I20260806 23:52:21.713721 21521 leader_election.cc:302] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b3078ff45f478a98e40232ca8dad04, 967fd0ceedd74c40aa33e18b1723df9a; no voters:
I20260806 23:52:21.713935 22584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 23:52:21.714010 22584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:21.714058 22584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 2 FOLLOWER]: Advancing to term 3
I20260806 23:52:21.714241 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.714006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53938)
0806 23:52:21.714143 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.714193 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 23:52:21.714349 21518 leader_election.cc:334] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319): Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
I20260806 23:52:21.715207 22584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } }
I20260806 23:52:21.715513 22584 leader_election.cc:288] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 3 election: Requested vote from peers 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319), 42b3078ff45f478a98e40232ca8dad04 (127.10.225.188:38663)
I20260806 23:52:21.715842 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.715616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53940)
0806 23:52:21.715732 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.715790 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 23:52:21.715893 21839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "967fd0ceedd74c40aa33e18b1723df9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42b3078ff45f478a98e40232ca8dad04"
I20260806 23:52:21.716022 21839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 2 FOLLOWER]: Advancing to term 3
I20260806 23:52:21.716456 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.716249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53954)
0806 23:52:21.716366 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.716412 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 23:52:21.716583 21518 leader_election.cc:334] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319): Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
I20260806 23:52:21.717249 21839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 967fd0ceedd74c40aa33e18b1723df9a in term 3.
I20260806 23:52:21.717770 21521 leader_election.cc:302] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42b3078ff45f478a98e40232ca8dad04, 967fd0ceedd74c40aa33e18b1723df9a; no voters: 8398b82fa36145bba7638367222ecd03
I20260806 23:52:21.718047 22584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 23:52:21.718214 22584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [term 3 LEADER]: Becoming Leader. State: Replica: 967fd0ceedd74c40aa33e18b1723df9a, State: Running, Role: LEADER
I20260806 23:52:21.718487 22584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } }
I20260806 23:52:21.719606 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 967fd0ceedd74c40aa33e18b1723df9a. Latest consensus state: current_term: 3 leader_uuid: "967fd0ceedd74c40aa33e18b1723df9a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:21.719758 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:21.720094 22590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:21.722752 22590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:21.723132 22590 catalog_manager.cc:1259] Loaded cluster ID: 97d8fd91848547a78de6f9001ce17931
I20260806 23:52:21.723176 22590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:21.723729 22590 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:21.724133 22590 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Loaded TSK: 0
I20260806 23:52:21.724810 22590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:52:21.772806 21540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:52:21.776222 21540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:21.779505 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.779118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53974)
0806 23:52:21.779302 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.779403 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 23:52:21.779531 21839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [term 3 FOLLOWER]: Refusing update from remote peer 967fd0ceedd74c40aa33e18b1723df9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 23:52:21.780275 22587 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 23:52:21.780700 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.780434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.0.0.1:53978)
0806 23:52:21.780587 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.780636 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260806 23:52:21.780817 21518 consensus_peers.cc:487] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a -> Peer 8398b82fa36145bba7638367222ecd03 (127.10.225.190:46319): Couldn't send request to peer 8398b82fa36145bba7638367222ecd03. Status: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 23:52:21.781900 22597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 967fd0ceedd74c40aa33e18b1723df9a. Latest consensus state: current_term: 3 leader_uuid: "967fd0ceedd74c40aa33e18b1723df9a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:21.782044 22597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:52:21.783385 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "967fd0ceedd74c40aa33e18b1723df9a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:21.783550 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:21.783977 22597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "967fd0ceedd74c40aa33e18b1723df9a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8398b82fa36145bba7638367222ecd03" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46319 } } peers { permanent_uuid: "967fd0ceedd74c40aa33e18b1723df9a" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 39079 } attrs { promote: false } } peers { permanent_uuid: "42b3078ff45f478a98e40232ca8dad04" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 38663 } attrs { promote: false } } }
I20260806 23:52:21.784083 22597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b3078ff45f478a98e40232ca8dad04 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:52:21.795295 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.791385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:33373 (local address 127.0.0.1:35896)
0806 23:52:21.791581 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.791591 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:21.791645 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:21.792305 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:21.792307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:21.792323 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:21.792406 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.792412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.793154 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.793157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.794280 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.794288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.794885 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.794889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.794972 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.794996 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:52:21.795062 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 23:52:21.795136 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260806 23:52:21.795433 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.790476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:44527 (local address 127.0.0.1:59618)
0806 23:52:21.790840 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.790908 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:21.790996 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:21.791759 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:21.791763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:21.791780 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:21.791883 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.791887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.792900 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.792904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.794179 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.794190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.795071 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.795074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.795155 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.795175 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 23:52:21.795243 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 23:52:21.795313 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":54}
I20260806 23:52:21.795859 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.791769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.10.225.129:33373)
0806 23:52:21.792095 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:21.792098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:21.792109 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:52:21.792153 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:21.792155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:21.792194 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:21.792270 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:21.792526 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.793054 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.794372 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.794779 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.794885 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.794944 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:21.795629 (+ 685us) server_negotiation.cc:299] Negotiation successful
0806 23:52:21.795742 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260806 23:52:21.795969 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.790994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:34931 (local address 127.0.0.1:56244)
0806 23:52:21.791253 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.791265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:21.791345 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:21.792410 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:21.792413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:21.792427 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:21.792510 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.792516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.793628 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.793631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.794619 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.794629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.795493 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.795496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.795577 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.795601 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:52:21.795642 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 23:52:21.795739 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
I20260806 23:52:21.796145 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.790637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59618 (local address 127.10.225.131:44527)
0806 23:52:21.791481 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:21.791487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:21.791501 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:21.791557 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:21.791560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:21.791612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:21.791727 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:21.791996 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.792752 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.794316 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.794920 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.795065 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.795117 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:21.795267 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 23:52:21.795357 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":87,"threads_started":1}
I20260806 23:52:21.796591 21991 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce1961af4d8442a9f4ab190834bfe59 (DEFAULT_TABLE table=test-table [id=f7b61d1e40b0417ca3548805c19e8545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:21.796651 22366 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce1961af4d8442a9f4ab190834bfe59 (DEFAULT_TABLE table=test-table [id=f7b61d1e40b0417ca3548805c19e8545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:21.796748 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.791171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.10.225.130:34931)
0806 23:52:21.792145 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:21.792150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:21.792162 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:52:21.792215 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:21.792217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:21.792264 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:21.792399 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:21.792607 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.793513 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.794768 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.795358 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.796435 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.796490 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:21.796533 (+ 43us) server_negotiation.cc:299] Negotiation successful
0806 23:52:21.796609 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":70,"threads_started":1}
I20260806 23:52:21.796999 21991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce1961af4d8442a9f4ab190834bfe59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:21.797032 22366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce1961af4d8442a9f4ab190834bfe59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:21.797313 22180 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce1961af4d8442a9f4ab190834bfe59 (DEFAULT_TABLE table=test-table [id=f7b61d1e40b0417ca3548805c19e8545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:21.797595 22180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce1961af4d8442a9f4ab190834bfe59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:21.799702 22610 tablet_bootstrap.cc:492] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: Bootstrap starting.
I20260806 23:52:21.800678 22610 tablet_bootstrap.cc:654] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:21.801481 22609 tablet_bootstrap.cc:492] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c: Bootstrap starting.
I20260806 23:52:21.802320 22609 tablet_bootstrap.cc:654] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:21.802512 22610 tablet_bootstrap.cc:492] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: No bootstrap required, opened a new log
I20260806 23:52:21.802656 22610 ts_tablet_manager.cc:1397] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:21.802927 22611 tablet_bootstrap.cc:492] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: Bootstrap starting.
I20260806 23:52:21.803196 22610 raft_consensus.cc:348] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:21.803326 22610 raft_consensus.cc:374] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:21.803347 22610 raft_consensus.cc:729] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bf272e79cc447db72a89c8e52aa8f9, State: Initialized, Role: FOLLOWER
I20260806 23:52:21.803479 22610 consensus_queue.cc:260] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:21.803807 22611 tablet_bootstrap.cc:654] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:21.804021 22610 ts_tablet_manager.cc:1428] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 23:52:21.805065 22609 tablet_bootstrap.cc:492] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c: No bootstrap required, opened a new log
I20260806 23:52:21.805148 22609 ts_tablet_manager.cc:1397] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 23:52:21.805403 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.805098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.131:53463)
0806 23:52:21.805236 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.805309 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 23:52:21.805570 22609 raft_consensus.cc:348] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:21.805688 22609 raft_consensus.cc:374] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:21.805716 22609 raft_consensus.cc:729] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07476883c4614a9c88ef41245bb4b35c, State: Initialized, Role: FOLLOWER
I20260806 23:52:21.805887 22609 consensus_queue.cc:260] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:21.806273 22609 ts_tablet_manager.cc:1428] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:21.807137 22611 tablet_bootstrap.cc:492] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: No bootstrap required, opened a new log
I20260806 23:52:21.807183 22464 heartbeater.cc:503] Master 127.10.225.189:39079 was elected leader, sending a full tablet report...
I20260806 23:52:21.807241 22611 ts_tablet_manager.cc:1397] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260806 23:52:21.807442 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.807147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.129:49765)
0806 23:52:21.807286 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.807350 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 23:52:21.807657 22611 raft_consensus.cc:348] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:21.807808 22611 raft_consensus.cc:374] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:21.807864 22611 raft_consensus.cc:729] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b80aa9e9c647a1af02c6718b38d4a3, State: Initialized, Role: FOLLOWER
I20260806 23:52:21.808024 22611 consensus_queue.cc:260] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:21.808403 22611 ts_tablet_manager.cc:1428] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 23:52:21.808696 22087 heartbeater.cc:503] Master 127.10.225.189:39079 was elected leader, sending a full tablet report...
I20260806 23:52:21.809494 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.809199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.130:41449)
0806 23:52:21.809321 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.809392 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 23:52:21.810237 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.809976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.130:49101)
0806 23:52:21.810109 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.810174 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 23:52:21.810602 22274 heartbeater.cc:503] Master 127.10.225.189:39079 was elected leader, sending a full tablet report...
I20260806 23:52:21.811820 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.811518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.129:48503)
0806 23:52:21.811649 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.811732 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 23:52:21.942875 22618 raft_consensus.cc:482] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:21.943066 22618 raft_consensus.cc:504] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:21.944314 22618 leader_election.cc:288] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9bf272e79cc447db72a89c8e52aa8f9 (127.10.225.131:44527), 07476883c4614a9c88ef41245bb4b35c (127.10.225.129:33373)
I20260806 23:52:21.948385 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.944654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:33373 (local address 127.10.225.130:50023)
0806 23:52:21.945085 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.945100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:21.945200 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:21.945423 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:21.945426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:21.945453 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:21.945583 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.945589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.946316 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.946319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.947249 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.947258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.947936 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.947940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.948042 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.948080 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 23:52:21.948145 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:52:21.948221 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":43,"spinlock_wait_cycles":896,"thread_start_us":129,"threads_started":1}
I20260806 23:52:21.948374 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.944764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:50023 (local address 127.10.225.129:33373)
0806 23:52:21.944904 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:21.944909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:21.945177 (+ 268us) server_negotiation.cc:407] Connection header received
0806 23:52:21.945289 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:21.945293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:21.945338 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:21.945400 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:21.945684 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.946199 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.947372 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.947824 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.947932 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.947991 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:21.948171 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 23:52:21.948254 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:52:21.948386 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.945005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:42005 (local address 127.10.225.131:44527)
0806 23:52:21.945154 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:21.945158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:21.945172 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:21.945249 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:21.945252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:21.945301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:21.945368 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:21.945680 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.946212 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.947380 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.947846 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.947954 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.948009 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:21.948197 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 23:52:21.948274 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 23:52:21.948407 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.944575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:44527 (local address 127.10.225.130:42005)
0806 23:52:21.945037 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.945052 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:21.945162 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:21.945386 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:21.945388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:21.945405 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:21.945557 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.945563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.946321 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.946332 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.947250 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:21.947260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.947954 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.947958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:21.948087 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.948105 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 23:52:21.948170 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:52:21.948241 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":82,"spinlock_wait_cycles":2432}
I20260806 23:52:21.948923 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce1961af4d8442a9f4ab190834bfe59" candidate_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" is_pre_election: true
I20260806 23:52:21.948987 22017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce1961af4d8442a9f4ab190834bfe59" candidate_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07476883c4614a9c88ef41245bb4b35c" is_pre_election: true
I20260806 23:52:21.949087 22400 raft_consensus.cc:2454] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8b80aa9e9c647a1af02c6718b38d4a3 in term 0.
I20260806 23:52:21.949240 22017 raft_consensus.cc:2454] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8b80aa9e9c647a1af02c6718b38d4a3 in term 0.
I20260806 23:52:21.949493 22116 leader_election.cc:302] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9bf272e79cc447db72a89c8e52aa8f9, f8b80aa9e9c647a1af02c6718b38d4a3; no voters:
I20260806 23:52:21.949812 22618 raft_consensus.cc:2781] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:21.949879 22618 raft_consensus.cc:482] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:21.949906 22618 raft_consensus.cc:3037] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:21.950992 22618 raft_consensus.cc:504] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:21.951346 22618 leader_election.cc:288] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [CANDIDATE]: Term 1 election: Requested vote from peers b9bf272e79cc447db72a89c8e52aa8f9 (127.10.225.131:44527), 07476883c4614a9c88ef41245bb4b35c (127.10.225.129:33373)
I20260806 23:52:21.951732 22017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce1961af4d8442a9f4ab190834bfe59" candidate_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07476883c4614a9c88ef41245bb4b35c"
I20260806 23:52:21.951853 22017 raft_consensus.cc:3037] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:21.951866 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce1961af4d8442a9f4ab190834bfe59" candidate_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bf272e79cc447db72a89c8e52aa8f9"
I20260806 23:52:21.951999 22400 raft_consensus.cc:3037] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:21.952920 22400 raft_consensus.cc:2454] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b80aa9e9c647a1af02c6718b38d4a3 in term 1.
I20260806 23:52:21.953212 22116 leader_election.cc:302] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9bf272e79cc447db72a89c8e52aa8f9, f8b80aa9e9c647a1af02c6718b38d4a3; no voters:
I20260806 23:52:21.953373 22618 raft_consensus.cc:2781] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:21.953438 22017 raft_consensus.cc:2454] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b80aa9e9c647a1af02c6718b38d4a3 in term 1.
I20260806 23:52:21.953622 22618 raft_consensus.cc:686] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 1 LEADER]: Becoming Leader. State: Replica: f8b80aa9e9c647a1af02c6718b38d4a3, State: Running, Role: LEADER
I20260806 23:52:21.953905 22618 consensus_queue.cc:237] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } }
I20260806 23:52:21.954389 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.954121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.130:45335)
0806 23:52:21.954236 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.954295 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 23:52:21.956900 21539 catalog_manager.cc:5495] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8b80aa9e9c647a1af02c6718b38d4a3 (127.10.225.130). New cstate: current_term: 1 leader_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8b80aa9e9c647a1af02c6718b38d4a3" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 34931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 } health_report { overall_health: UNKNOWN } } }
I20260806 23:52:21.992257 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.988689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.10.225.130:34931)
0806 23:52:21.988824 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:21.988827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:21.988842 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:52:21.988961 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:21.988964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:21.989021 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:21.989111 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:21.989346 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.989862 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.990734 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:21.990956 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:21.991047 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:21.991310 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:21.991859 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:21.991949 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:21.992049 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 23:52:21.992106 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:52:21.995216 22400 raft_consensus.cc:1261] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [term 1 FOLLOWER]: Refusing update from remote peer f8b80aa9e9c647a1af02c6718b38d4a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:21.995468 22017 raft_consensus.cc:1261] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [term 1 FOLLOWER]: Refusing update from remote peer f8b80aa9e9c647a1af02c6718b38d4a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:21.995769 22621 consensus_queue.cc:1035] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bf272e79cc447db72a89c8e52aa8f9" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:21.996093 22618 consensus_queue.cc:1035] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07476883c4614a9c88ef41245bb4b35c" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:21.998168 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.997277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.131:52167)
0806 23:52:21.998019 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.998085 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652}
I20260806 23:52:21.998495 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:21.998075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.129:41287)
0806 23:52:21.998315 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:21.998386 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260806 23:52:22.002197 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:22.001884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.129:36529)
0806 23:52:22.002045 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:22.002107 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 23:52:22.003790 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:22.003454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.131:33781)
0806 23:52:22.003617 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:22.003699 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 23:52:22.004020 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:22.003530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46319 (local address 127.10.225.130:59315)
0806 23:52:22.003880 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:22.003936 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:46319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280}
I20260806 23:52:22.004725 22635 mvcc.cc:205] Tried to move back new op lower bound from 7315703160807075840 to 7315703160646156288. Current Snapshot: MvccSnapshot[applied={T|T < 7315703160807075840}]
I20260806 23:52:22.014202 21542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:22.014434 21542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:22.019389 21542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 967fd0ceedd74c40aa33e18b1723df9a: Sending DeleteTablet for 3 replicas of tablet 2ce1961af4d8442a9f4ab190834bfe59
I20260806 23:52:22.020179 22180 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce1961af4d8442a9f4ab190834bfe59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:22 UTC) from {username='oryx', principal='oryx/127.10.225.189@KRBTEST.COM'} at 127.0.0.1:56244
I20260806 23:52:22.020365 21991 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce1961af4d8442a9f4ab190834bfe59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:22 UTC) from {username='oryx', principal='oryx/127.10.225.189@KRBTEST.COM'} at 127.0.0.1:35896
I20260806 23:52:22.020378 22366 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce1961af4d8442a9f4ab190834bfe59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:22 UTC) from {username='oryx', principal='oryx/127.10.225.189@KRBTEST.COM'} at 127.0.0.1:59618
I20260806 23:52:22.020582 22644 tablet_replica.cc:330] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: stopping tablet replica
I20260806 23:52:22.020756 22644 raft_consensus.cc:2229] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:52:22.020949 22647 tablet_replica.cc:330] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c: stopping tablet replica
I20260806 23:52:22.021124 22647 raft_consensus.cc:2229] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:22.021145 22644 raft_consensus.cc:2258] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:22.021324 22647 raft_consensus.cc:2258] T 2ce1961af4d8442a9f4ab190834bfe59 P 07476883c4614a9c88ef41245bb4b35c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:22.021382 22648 tablet_replica.cc:330] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: stopping tablet replica
I20260806 23:52:22.021518 22648 raft_consensus.cc:2229] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:22.021759 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 21770
I20260806 23:52:22.021657 22648 raft_consensus.cc:2258] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:22.025094 22648 ts_tablet_manager.cc:1905] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:22.028025 22648 ts_tablet_manager.cc:1918] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:22.028117 22648 log.cc:1196] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/2ce1961af4d8442a9f4ab190834bfe59
I20260806 23:52:22.028422 22648 ts_tablet_manager.cc:1939] T 2ce1961af4d8442a9f4ab190834bfe59 P b9bf272e79cc447db72a89c8e52aa8f9: Deleting consensus metadata
I20260806 23:52:22.029538 21520 catalog_manager.cc:4841] TS b9bf272e79cc447db72a89c8e52aa8f9 (127.10.225.131:44527): tablet 2ce1961af4d8442a9f4ab190834bfe59 (table test-table [id=f7b61d1e40b0417ca3548805c19e8545]) successfully deleted
I20260806 23:52:22.051956 22644 ts_tablet_manager.cc:1905] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:22.054795 22644 ts_tablet_manager.cc:1918] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:22.054885 22644 log.cc:1196] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/2ce1961af4d8442a9f4ab190834bfe59
I20260806 23:52:22.055119 22644 ts_tablet_manager.cc:1939] T 2ce1961af4d8442a9f4ab190834bfe59 P f8b80aa9e9c647a1af02c6718b38d4a3: Deleting consensus metadata
I20260806 23:52:22.055927 21521 catalog_manager.cc:4841] TS f8b80aa9e9c647a1af02c6718b38d4a3 (127.10.225.130:34931): tablet 2ce1961af4d8442a9f4ab190834bfe59 (table test-table [id=f7b61d1e40b0417ca3548805c19e8545]) successfully deleted
I20260806 23:52:22.063645 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 22096
I20260806 23:52:22.064005 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:22.063560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:33373 (local address 127.0.0.1:35898)
0806 23:52:22.063785 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:22.063877 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:33373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260806 23:52:22.071237 21518 catalog_manager.cc:4584] TS 07476883c4614a9c88ef41245bb4b35c (127.10.225.129:33373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ce1961af4d8442a9f4ab190834bfe59: Network error: Client connection negotiation failed: client connection to 127.10.225.129:33373: connect: Connection refused (error 111)
I20260806 23:52:22.081107 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 22282
I20260806 23:52:22.093978 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 21335
I20260806 23:52:22.108862 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 21486
2026-08-06T23:52:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7684 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:52:22 dist-test-slave-nj21 krb5kdc[22653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:52:22 dist-test-slave-nj21 krb5kdc[22653](info): set up 2 sockets
Aug 06 23:52:22 dist-test-slave-nj21 krb5kdc[22653](info): commencing operation
krb5kdc: starting...
W20260806 23:52:23.303823 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:23 dist-test-slave-nj21 krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.190.keytab.
Entry for principal oryx/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.190.keytab.
I20260806 23:52:23.447216 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35377
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.190.keytab
--principal=oryx/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:23.573976 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:23.574287 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:23.574349 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:23.578868 22672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:23.578975 22672 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 23:52:23.579002 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:23.579030 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:23.579053 22672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:23.579088 22672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:23.585130 22672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35377
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.190.keytab
--principal=oryx/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:23.586350 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:23.587463 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:23.596889 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:23.596890 22683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:23.596889 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:23.596979 22672 server_base.cc:1034] running on GCE node
I20260806 23:52:23.597718 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:23.598897 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:23.600068 22672 hybrid_clock.cc:648] HybridClock initialized: now 1786060343600051 us; error 30 us; skew 500 ppm
Aug 06 23:52:23 dist-test-slave-nj21 krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060343, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:23.603037 22672 init.cc:377] Logged in from keytab as oryx/127.10.225.190@KRBTEST.COM (short username oryx)
I20260806 23:52:23.604110 22672 webserver.cc:460] Webserver started at http://127.10.225.190:34603/ using document root <none> and password file <none>
I20260806 23:52:23.604681 22672 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:23.604743 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:23.604943 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:23.607164 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "98507f57d3364114932cdf9a9d5a7920"
format_stamp: "Formatted at 2026-08-06 23:52:23 on dist-test-slave-nj21"
server_key: "275e821e335a328cf0ebd6cd61c791f3"
server_key_iv: "22de782f43421cb6fb804e975fb9a836"
server_key_version: "encryptionkey@0"
I20260806 23:52:23.607800 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "98507f57d3364114932cdf9a9d5a7920"
format_stamp: "Formatted at 2026-08-06 23:52:23 on dist-test-slave-nj21"
server_key: "275e821e335a328cf0ebd6cd61c791f3"
server_key_iv: "22de782f43421cb6fb804e975fb9a836"
server_key_version: "encryptionkey@0"
I20260806 23:52:23.612092 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:52:23.614950 22693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:23.616968 22672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:23.617148 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "98507f57d3364114932cdf9a9d5a7920"
format_stamp: "Formatted at 2026-08-06 23:52:23 on dist-test-slave-nj21"
server_key: "275e821e335a328cf0ebd6cd61c791f3"
server_key_iv: "22de782f43421cb6fb804e975fb9a836"
server_key_version: "encryptionkey@0"
I20260806 23:52:23.617271 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:23.633127 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:23.636394 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:23.636600 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:23.644572 22672 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:35377
I20260806 23:52:23.644590 22776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:35377 every 8 connection(s)
I20260806 23:52:23.645627 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:23.646878 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 22672
I20260806 23:52:23.647027 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:23.647419 11142 external_mini_cluster.cc:1428] Setting key 0d74a834197018a6dac1fce74bedbbd9
I20260806 23:52:23.649598 22777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 23:52:23 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:23.657869 22777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: Bootstrap starting.
I20260806 23:52:23.660329 22777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:23.661778 22777 log.cc:824] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:23.662919 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:23.648917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.10.225.190:35377)
0806 23:52:23.649338 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:23.649353 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:23.649388 (+ 35us) server_negotiation.cc:407] Connection header received
0806 23:52:23.650072 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:23.650089 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:23.650418 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:23.650690 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:23.651751 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:23.652772 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:23.654292 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:23.654804 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:23.654952 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:23.658180 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:23.658185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:23.658199 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:23.658230 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:23.659888 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:23.660434 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:23.660441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:23.660448 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:23.660601 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:23.660933 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:23.660935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:23.660936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:23.661354 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:23.661509 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:23.661848 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 23:52:23.662165 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":119,"threads_started":1}
I20260806 23:52:23.663964 22777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: No bootstrap required, opened a new log
I20260806 23:52:23.667239 22777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98507f57d3364114932cdf9a9d5a7920" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35377 } }
I20260806 23:52:23.667480 22777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:23.667531 22777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98507f57d3364114932cdf9a9d5a7920, State: Initialized, Role: FOLLOWER
I20260806 23:52:23.667975 22777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98507f57d3364114932cdf9a9d5a7920" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35377 } }
I20260806 23:52:23.668114 22777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:23.668166 22777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:23.668248 22777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:23.669226 22777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98507f57d3364114932cdf9a9d5a7920" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35377 } }
I20260806 23:52:23.669868 22777 leader_election.cc:302] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98507f57d3364114932cdf9a9d5a7920; no voters:
I20260806 23:52:23.670425 22777 leader_election.cc:288] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:23.670619 22782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:23.670859 22782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [term 1 LEADER]: Becoming Leader. State: Replica: 98507f57d3364114932cdf9a9d5a7920, State: Running, Role: LEADER
I20260806 23:52:23.671190 22782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98507f57d3364114932cdf9a9d5a7920" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35377 } }
I20260806 23:52:23.671890 22777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:23.673954 22783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98507f57d3364114932cdf9a9d5a7920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98507f57d3364114932cdf9a9d5a7920" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35377 } } }
I20260806 23:52:23.674096 22783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:23.674057 22784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98507f57d3364114932cdf9a9d5a7920. Latest consensus state: current_term: 1 leader_uuid: "98507f57d3364114932cdf9a9d5a7920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98507f57d3364114932cdf9a9d5a7920" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35377 } } }
I20260806 23:52:23.674460 22784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920 [sys.catalog]: This master's current role is: LEADER
W20260806 23:52:23.678494 22812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:52:23.678579 22812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:52:23.678640 22811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:23.679525 22811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:23.689656 22811 catalog_manager.cc:1347] Generated new cluster ID: 18426f7585e84a5cbe114a29b3e79586
I20260806 23:52:23.689793 22811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:23.729187 22811 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:23.730248 22811 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:23.741683 22811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: Generated new TSK 0
I20260806 23:52:23.742506 22811 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab.
Entry for principal oryx/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab.
I20260806 23:52:23.787628 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35377
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab
--principal=oryx/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:23.905791 22819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:23.906225 22819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:23.906301 22819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:23.911536 22819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:23.911665 22819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:23.911791 22819 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:52:23.911859 22819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:23.918967 22819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab
--principal=oryx/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.225.190:35377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:23.920217 22819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:23.921368 22819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:23.931560 22826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:23.931480 22829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:23.931540 22819 server_base.cc:1034] running on GCE node
W20260806 23:52:23.931499 22825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:23.932178 22819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:23.932910 22819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:23.934090 22819 hybrid_clock.cc:648] HybridClock initialized: now 1786060343934060 us; error 43 us; skew 500 ppm
Aug 06 23:52:23 dist-test-slave-nj21 krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060343, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:23.937654 22819 init.cc:377] Logged in from keytab as oryx/127.10.225.129@KRBTEST.COM (short username oryx)
I20260806 23:52:23.938655 22819 webserver.cc:460] Webserver started at http://127.10.225.129:46641/ using document root <none> and password file <none>
I20260806 23:52:23.939240 22819 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:23.939308 22819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:23.939491 22819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:23.941779 22819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a"
format_stamp: "Formatted at 2026-08-06 23:52:23 on dist-test-slave-nj21"
server_key: "a19fead086ac350860d98c40bd501b91"
server_key_iv: "1a2d489e2907fdd6be9508033ec15650"
server_key_version: "encryptionkey@0"
I20260806 23:52:23.942353 22819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a"
format_stamp: "Formatted at 2026-08-06 23:52:23 on dist-test-slave-nj21"
server_key: "a19fead086ac350860d98c40bd501b91"
server_key_iv: "1a2d489e2907fdd6be9508033ec15650"
server_key_version: "encryptionkey@0"
I20260806 23:52:23.947178 22819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 23:52:23.950551 22839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:23.952314 22819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:23.952436 22819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a"
format_stamp: "Formatted at 2026-08-06 23:52:23 on dist-test-slave-nj21"
server_key: "a19fead086ac350860d98c40bd501b91"
server_key_iv: "1a2d489e2907fdd6be9508033ec15650"
server_key_version: "encryptionkey@0"
I20260806 23:52:23.952550 22819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:23.973568 22819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:23.977874 22819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:23.978063 22819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:23.980665 22819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:23.980753 22819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:23.980829 22819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:23.980860 22819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:52:23 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060343, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:23.994545 22819 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:46319
I20260806 23:52:23.995035 22991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:46319 every 8 connection(s)
I20260806 23:52:23.995713 22819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:23.996555 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 22819
I20260806 23:52:23.996681 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:23.997042 11142 external_mini_cluster.cc:1428] Setting key 8bb5c0faac861f224af3a66a977a31bb
I20260806 23:52:23.998719 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:23.981465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:34705 (local address 127.10.225.190:35377)
0806 23:52:23.981695 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:23.981703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:23.982837 (+ 1134us) server_negotiation.cc:407] Connection header received
0806 23:52:23.984275 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:23.984279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:23.984358 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:23.984464 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:23.986525 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:23.987414 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:23.989850 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:23.990162 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:23.990307 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:23.993806 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:23.993815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:23.993818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:23.993853 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:23.995521 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:23.996390 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:23.996394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:23.996396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:23.996497 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:23.997158 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:23.997164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:23.997166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:23.997515 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:23.997680 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:23.998363 (+ 683us) server_negotiation.cc:299] Negotiation successful
0806 23:52:23.998532 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 23:52:23.999707 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:23.981978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35377 (local address 127.10.225.129:34705)
0806 23:52:23.982599 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:23.982663 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:23.983925 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:23.984786 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:23.984800 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:23.985382 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:23.986193 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:23.986234 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:23.987736 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:23.987742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:23.989658 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:23.989678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:23.990393 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:23.990399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:23.990615 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:23.991287 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:23.991308 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:23.993382 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:23.995649 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:23.995656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:23.995710 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:23.996223 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:23.996611 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:23.996614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:23.996616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:23.996968 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:23.997657 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:23.997665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:23.997983 (+ 318us) client_negotiation.cc:769] Sending connection context
0806 23:52:23.998204 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 23:52:23.998591 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":153,"threads_started":1}
I20260806 23:52:24.002310 22992 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35377
I20260806 23:52:24.002492 22992 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:24.003118 22992 heartbeater.cc:511] Master 127.10.225.190:35377 requested a full tablet report, sending...
I20260806 23:52:24.004918 22714 ts_manager.cc:194] Registered new tserver with Master: bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319)
I20260806 23:52:24.006269 22714 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:34705
WARNING: no policy specified for oryx/127.10.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.130@KRBTEST.COM" created.
I20260806 23:52:24.013430 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.004725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35377 (local address 127.10.225.129:45141)
0806 23:52:24.005095 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.005110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.005221 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.005465 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.005469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.005712 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:24.005858 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.005866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.006827 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.006831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.007927 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.007941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.008540 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.008545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.008650 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.009191 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:24.009208 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:24.009633 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:24.011663 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.011666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.011668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.011947 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.012308 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.012311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.012313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.012437 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.012924 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:24.012927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:24.013021 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.013114 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.013240 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
I20260806 23:52:24.013491 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.004643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:45141 (local address 127.10.225.190:35377)
0806 23:52:24.004777 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.004781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.005187 (+ 406us) server_negotiation.cc:407] Connection header received
0806 23:52:24.005302 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.005305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.005350 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.005431 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:24.005983 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.006692 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.008080 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.008397 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.008503 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.009807 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:24.009810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:24.009813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:24.009835 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:24.011545 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.012087 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.012091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.012093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.012205 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.012533 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.012537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.012539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.012795 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:24.012901 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.013184 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.013322 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab.
Entry for principal oryx/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab.
I20260806 23:52:24.031639 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35377
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab
--principal=oryx/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:24.157841 23004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:24.158154 23004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:24.158224 23004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:24.162698 23004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:24.162799 23004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:24.162871 23004 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:52:24.162945 23004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:24.168895 23004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab
--principal=oryx/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.225.190:35377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:24.170163 23004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:24.171260 23004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:24.181697 23012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:24.181700 23015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:24.181733 23011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:24.182118 23004 server_base.cc:1034] running on GCE node
I20260806 23:52:24.182585 23004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:24.183270 23004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:24.184450 23004 hybrid_clock.cc:648] HybridClock initialized: now 1786060344184434 us; error 31 us; skew 500 ppm
Aug 06 23:52:24 dist-test-slave-nj21 krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060344, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:24.187427 23004 init.cc:377] Logged in from keytab as oryx/127.10.225.130@KRBTEST.COM (short username oryx)
I20260806 23:52:24.188438 23004 webserver.cc:460] Webserver started at http://127.10.225.130:37571/ using document root <none> and password file <none>
I20260806 23:52:24.189023 23004 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:24.189093 23004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:24.189309 23004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:24.191946 23004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8"
format_stamp: "Formatted at 2026-08-06 23:52:24 on dist-test-slave-nj21"
server_key: "19ccc1f667709c5ac895147f9f2ec7a6"
server_key_iv: "e476e87393e698a6c3940369c9746988"
server_key_version: "encryptionkey@0"
I20260806 23:52:24.192533 23004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8"
format_stamp: "Formatted at 2026-08-06 23:52:24 on dist-test-slave-nj21"
server_key: "19ccc1f667709c5ac895147f9f2ec7a6"
server_key_iv: "e476e87393e698a6c3940369c9746988"
server_key_version: "encryptionkey@0"
I20260806 23:52:24.196727 23004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:52:24.199332 23023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:24.200691 23004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:24.200810 23004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8"
format_stamp: "Formatted at 2026-08-06 23:52:24 on dist-test-slave-nj21"
server_key: "19ccc1f667709c5ac895147f9f2ec7a6"
server_key_iv: "e476e87393e698a6c3940369c9746988"
server_key_version: "encryptionkey@0"
I20260806 23:52:24.200927 23004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:24.220382 23004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:24.224805 23004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:24.225118 23004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:24.227164 23004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:24.227228 23004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:24.227294 23004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:24.227322 23004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:52:24 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060344, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:24.238539 23004 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:35841
I20260806 23:52:24.238643 23179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:35841 every 8 connection(s)
I20260806 23:52:24.239559 23004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:24.241420 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 23004
I20260806 23:52:24.241539 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:24.241873 11142 external_mini_cluster.cc:1428] Setting key 33e6ebdc4d5ab670e2bf3e55b504ed8c
I20260806 23:52:24.243366 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.228331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:56277 (local address 127.10.225.190:35377)
0806 23:52:24.228461 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.228465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.229892 (+ 1427us) server_negotiation.cc:407] Connection header received
0806 23:52:24.231351 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.231355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.231432 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.231523 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:24.233141 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.233738 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.235516 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.235829 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.235941 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.239203 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:24.239207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:24.239209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:24.239241 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:24.241069 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.241650 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.241653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.241655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.241734 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.242078 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.242081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.242083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.242348 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:24.242453 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.243058 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.243194 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 23:52:24.244493 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.228946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35377 (local address 127.10.225.130:56277)
0806 23:52:24.229654 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.229715 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.231057 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.231809 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.231818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.232333 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:24.232964 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.232992 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.233872 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.233876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.235286 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.235316 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.236005 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.236009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.236330 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.237083 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:24.237101 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:24.238922 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:24.241187 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.241193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.241205 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.241550 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.241825 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.241828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.241831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.241987 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.242479 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:24.242487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:24.242815 (+ 328us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.242999 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.243362 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":179,"threads_started":1}
I20260806 23:52:24.245766 23180 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35377
I20260806 23:52:24.245997 23180 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:24.246614 23180 heartbeater.cc:511] Master 127.10.225.190:35377 requested a full tablet report, sending...
I20260806 23:52:24.247882 22716 ts_manager.cc:194] Registered new tserver with Master: 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841)
I20260806 23:52:24.248667 22716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:56277
WARNING: no policy specified for oryx/127.10.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.131@KRBTEST.COM" created.
I20260806 23:52:24.260874 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.248195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35377 (local address 127.10.225.130:43321)
0806 23:52:24.253628 (+ 5433us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.253647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.253776 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.254060 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.254062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.254273 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:24.254389 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.254396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.255247 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.255251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.255949 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.255960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.256500 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.256504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.256598 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.257128 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:24.257145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:24.257580 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:24.259211 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.259215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.259217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.259427 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.259757 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.259761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.259763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.259868 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.260384 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:24.260386 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:24.260480 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.260564 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.260699 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5175,"thread_start_us":359,"threads_started":1}
I20260806 23:52:24.260957 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.248261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:43321 (local address 127.10.225.190:35377)
0806 23:52:24.248584 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.248589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.253762 (+ 5173us) server_negotiation.cc:407] Connection header received
0806 23:52:24.253877 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.253879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.253936 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.254026 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:24.254530 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.255102 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.256110 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.256375 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.256515 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.257738 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:24.257742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:24.257744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:24.257768 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:24.259103 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.259520 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.259524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.259525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.259612 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.259966 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.259969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.259971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.260269 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:24.260372 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.260656 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.260780 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab.
Entry for principal oryx/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab.
I20260806 23:52:24.275245 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35377
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab
--principal=oryx/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:24.393893 23190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:24.394184 23190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:24.394230 23190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:24.398444 23190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:24.398530 23190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:24.398572 23190 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 23:52:24.398607 23190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:24.405915 23190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab
--principal=oryx/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.225.190:35377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:24.407526 23190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:24.408977 23190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:24.420480 23197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:24.420501 23198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:24.420476 23200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:24.420871 23190 server_base.cc:1034] running on GCE node
I20260806 23:52:24.421329 23190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:24.422035 23190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:24.423223 23190 hybrid_clock.cc:648] HybridClock initialized: now 1786060344423182 us; error 61 us; skew 500 ppm
Aug 06 23:52:24 dist-test-slave-nj21 krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060344, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:24.426174 23190 init.cc:377] Logged in from keytab as oryx/127.10.225.131@KRBTEST.COM (short username oryx)
I20260806 23:52:24.427175 23190 webserver.cc:460] Webserver started at http://127.10.225.131:34667/ using document root <none> and password file <none>
I20260806 23:52:24.427783 23190 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:24.427853 23190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:24.428049 23190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:24.430181 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "aab8dac7f6de4764ac6c407661feccef"
format_stamp: "Formatted at 2026-08-06 23:52:24 on dist-test-slave-nj21"
server_key: "002f9efc31c4be88ed5b89d7d02ba573"
server_key_iv: "82c7719a1b42860299b9ec573896c656"
server_key_version: "encryptionkey@0"
I20260806 23:52:24.430707 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "aab8dac7f6de4764ac6c407661feccef"
format_stamp: "Formatted at 2026-08-06 23:52:24 on dist-test-slave-nj21"
server_key: "002f9efc31c4be88ed5b89d7d02ba573"
server_key_iv: "82c7719a1b42860299b9ec573896c656"
server_key_version: "encryptionkey@0"
I20260806 23:52:24.434746 23190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 23:52:24.437592 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:24.439754 23190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:24.439989 23190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "aab8dac7f6de4764ac6c407661feccef"
format_stamp: "Formatted at 2026-08-06 23:52:24 on dist-test-slave-nj21"
server_key: "002f9efc31c4be88ed5b89d7d02ba573"
server_key_iv: "82c7719a1b42860299b9ec573896c656"
server_key_version: "encryptionkey@0"
I20260806 23:52:24.440176 23190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:24.458662 23190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:24.462854 23190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:24.463069 23190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:24.464803 23190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:24.464860 23190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:24.464903 23190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:24.464918 23190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 23:52:24 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060344, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:24.475993 23190 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:46121
I20260806 23:52:24.476096 23369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:46121 every 8 connection(s)
I20260806 23:52:24.477015 23190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:24.478884 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.465754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:45045 (local address 127.10.225.190:35377)
0806 23:52:24.465954 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.465958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.466987 (+ 1029us) server_negotiation.cc:407] Connection header received
0806 23:52:24.468090 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.468093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.468153 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.468238 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:24.469955 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.470651 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.471953 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.472220 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.472314 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.475190 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:24.475193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:24.475195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:24.475222 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:24.476724 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.477346 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.477350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.477351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.477465 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.477865 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.477869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.477870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.478073 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:24.478153 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.478643 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.478760 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 23:52:24.479629 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.466277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35377 (local address 127.10.225.131:45045)
0806 23:52:24.466822 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.466866 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.467855 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.468471 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.468481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.468978 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:24.469757 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.469781 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.470773 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.470776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.471840 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.471852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.472381 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.472385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.472578 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.473254 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:24.473279 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:24.475009 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:24.476839 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.476845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.476860 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.477205 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.477593 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.477596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.477598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.477771 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.478150 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:24.478155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:24.478401 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.478558 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.478823 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":147,"threads_started":1}
I20260806 23:52:24.480804 23370 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35377
I20260806 23:52:24.481053 23370 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:24.481637 23370 heartbeater.cc:511] Master 127.10.225.190:35377 requested a full tablet report, sending...
I20260806 23:52:24.483120 22716 ts_manager.cc:194] Registered new tserver with Master: aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121)
I20260806 23:52:24.483721 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 23190
I20260806 23:52:24.483871 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:24.484104 22716 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:45045
I20260806 23:52:24.484227 11142 external_mini_cluster.cc:1428] Setting key 2a05b4d61bee94a2c771a3fdfa018f59
I20260806 23:52:24.486841 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:52:24.492136 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.483551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:59169 (local address 127.10.225.190:35377)
0806 23:52:24.483702 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.483706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.484179 (+ 473us) server_negotiation.cc:407] Connection header received
0806 23:52:24.484471 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.484474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.484524 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.484575 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:24.485301 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.485997 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.487011 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.487319 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.487394 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.488837 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:24.488840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:24.488843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:24.488866 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:24.490469 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.490888 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.490892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.490893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.490970 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.491301 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.491305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.491306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.491506 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:24.491590 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.491899 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.492006 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:52:24.492198 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.483473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35377 (local address 127.10.225.131:59169)
0806 23:52:24.484055 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.484074 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.484333 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.484715 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.484717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.484967 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:24.485106 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.485115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.486132 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.486134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.486836 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.486847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.487579 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.487582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.487705 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.488204 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:24.488218 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:24.488662 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:24.490588 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.490592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.490593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.490792 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.491072 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.491075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.491077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.491175 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.491605 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:24.491608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:24.491738 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.491855 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.492016 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":218,"threads_started":1}
Aug 06 23:52:24 dist-test-slave-nj21 krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060344, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 23:52:24 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:24 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 23:52:24.516248 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:24.508801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.10.225.190:35377)
0806 23:52:24.508942 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.508945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.508958 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:52:24.508998 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.509000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.509039 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.509101 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:24.509875 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.510658 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.511553 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.511873 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.511956 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.516106 (+ 4150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59534: BlockingRecv error: recv got EOF from 127.0.0.1:59534 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 23:52:24 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060344, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:24.528179 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.518912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.10.225.190:35377)
0806 23:52:24.519062 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.519067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.519142 (+ 75us) server_negotiation.cc:407] Connection header received
0806 23:52:24.519278 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.519281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.519326 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.519397 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:24.520119 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.520709 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.521558 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.521768 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.521856 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.524872 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:24.524877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:24.524880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:24.524923 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:24.526450 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.526867 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.526870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.526871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.526950 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.527305 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.527308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.527310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.527527 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:24.527644 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.527936 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.528046 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 23:52:24.531935 22716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:52:24.536044 22716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:24.553491 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.548235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.10.225.131:46121)
0806 23:52:24.548556 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.548567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.548589 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:52:24.548660 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.548665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.548834 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.549260 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:24.549397 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.550180 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.552213 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.552795 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.552929 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.553082 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.553221 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.553335 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":120,"threads_started":1}
I20260806 23:52:24.553501 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.547112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:46121 (local address 127.0.0.1:46596)
0806 23:52:24.548087 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.548101 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.548192 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.548998 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.549002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.549016 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:24.549127 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.549133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.550290 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.550293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.552067 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.552078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.552915 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.552919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.553025 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.553065 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.553181 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.553319 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":112,"thread_start_us":127,"threads_started":1}
I20260806 23:52:24.553531 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.547764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:46319 (local address 127.0.0.1:53458)
0806 23:52:24.548469 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.548483 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.548574 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.549280 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.549283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.549298 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:24.549402 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.549408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.550671 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.550675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.552026 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.552037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.553101 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.553105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.553205 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.553229 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.553280 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.553365 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":236,"thread_start_us":119,"threads_started":1}
I20260806 23:52:24.553853 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.547054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.10.225.130:35841)
0806 23:52:24.548147 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.548168 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.548198 (+ 30us) server_negotiation.cc:407] Connection header received
0806 23:52:24.548403 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.548411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.548645 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.548913 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:24.549186 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.550400 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.552174 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.552960 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.553107 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.553236 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.553598 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.553701 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":153,"threads_started":1}
I20260806 23:52:24.553898 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.546842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35841 (local address 127.0.0.1:54088)
0806 23:52:24.547542 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.547597 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.547774 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.548849 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.548856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.548879 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:24.549077 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.549085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.550542 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.550545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.552067 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.552078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.553295 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.553299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.553382 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.553404 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.553466 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.553524 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":161,"thread_start_us":135,"threads_started":1}
I20260806 23:52:24.554543 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.547933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.10.225.129:46319)
0806 23:52:24.548860 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.548877 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.548904 (+ 27us) server_negotiation.cc:407] Connection header received
0806 23:52:24.548979 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.548984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.549149 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.549297 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:24.549534 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.550571 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.552144 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.552960 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.554165 (+ 1205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.554261 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.554304 (+ 43us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.554406 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":113,"threads_started":1}
I20260806 23:52:24.557375 23275 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3d9b8d88e148daa773b168908600c1 (DEFAULT_TABLE table=test-table [id=82850a0c08b840eca6285a0bdbacb1d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:24.557737 23091 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3d9b8d88e148daa773b168908600c1 (DEFAULT_TABLE table=test-table [id=82850a0c08b840eca6285a0bdbacb1d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:24.557742 22903 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3d9b8d88e148daa773b168908600c1 (DEFAULT_TABLE table=test-table [id=82850a0c08b840eca6285a0bdbacb1d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:24.558537 23275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3d9b8d88e148daa773b168908600c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:24.558830 23091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3d9b8d88e148daa773b168908600c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:24.558827 22903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3d9b8d88e148daa773b168908600c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:24.565644 23405 tablet_bootstrap.cc:492] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: Bootstrap starting.
I20260806 23:52:24.566963 23408 tablet_bootstrap.cc:492] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Bootstrap starting.
I20260806 23:52:24.568019 23407 tablet_bootstrap.cc:492] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: Bootstrap starting.
I20260806 23:52:24.568356 23405 tablet_bootstrap.cc:654] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:24.569005 23408 tablet_bootstrap.cc:654] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:24.569103 23405 log.cc:824] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:24.569893 23408 log.cc:824] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:24.570192 23407 tablet_bootstrap.cc:654] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:24.571034 23407 log.cc:824] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:24.571590 23405 tablet_bootstrap.cc:492] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: No bootstrap required, opened a new log
I20260806 23:52:24.571841 23405 ts_tablet_manager.cc:1397] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:52:24.571892 23408 tablet_bootstrap.cc:492] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: No bootstrap required, opened a new log
I20260806 23:52:24.572086 23408 ts_tablet_manager.cc:1397] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:52:24.573145 23407 tablet_bootstrap.cc:492] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: No bootstrap required, opened a new log
I20260806 23:52:24.573341 23407 ts_tablet_manager.cc:1397] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:52:24.575205 23405 raft_consensus.cc:348] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } }
I20260806 23:52:24.575440 23405 raft_consensus.cc:374] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:24.575495 23405 raft_consensus.cc:729] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab8dac7f6de4764ac6c407661feccef, State: Initialized, Role: FOLLOWER
I20260806 23:52:24.575953 23405 consensus_queue.cc:260] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } }
I20260806 23:52:24.576200 23408 raft_consensus.cc:348] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } }
I20260806 23:52:24.576449 23408 raft_consensus.cc:374] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:24.576507 23408 raft_consensus.cc:729] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec5831e2e0745a2b52a7e48ae1a9ae8, State: Initialized, Role: FOLLOWER
I20260806 23:52:24.576963 23408 consensus_queue.cc:260] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } }
I20260806 23:52:24.577077 23405 ts_tablet_manager.cc:1428] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260806 23:52:24.577469 23370 heartbeater.cc:503] Master 127.10.225.190:35377 was elected leader, sending a full tablet report...
I20260806 23:52:24.577910 23408 ts_tablet_manager.cc:1428] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:52:24.577880 23407 raft_consensus.cc:348] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } }
I20260806 23:52:24.578135 23407 raft_consensus.cc:374] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:24.578187 23407 raft_consensus.cc:729] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3670a2c69e4ffeadf61d88bf1ac51a, State: Initialized, Role: FOLLOWER
I20260806 23:52:24.578262 23180 heartbeater.cc:503] Master 127.10.225.190:35377 was elected leader, sending a full tablet report...
I20260806 23:52:24.578636 23407 consensus_queue.cc:260] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } }
I20260806 23:52:24.579346 22992 heartbeater.cc:503] Master 127.10.225.190:35377 was elected leader, sending a full tablet report...
I20260806 23:52:24.579427 23407 ts_tablet_manager.cc:1428] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 23:52:24.612609 23416 raft_consensus.cc:482] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:24.612949 23416 raft_consensus.cc:504] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } }
I20260806 23:52:24.614507 23416 leader_election.cc:288] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121), bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319)
I20260806 23:52:24.619326 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.614912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:46319 (local address 127.10.225.130:55513)
0806 23:52:24.615159 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.615174 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.615357 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.615649 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.615651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.615692 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:24.615847 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.615854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.616776 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.616778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.618124 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.618141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.618893 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.618895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.618996 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.619027 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.619098 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.619180 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":40}
I20260806 23:52:24.619357 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.615028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:55513 (local address 127.10.225.129:46319)
0806 23:52:24.615150 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.615155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.615258 (+ 103us) server_negotiation.cc:407] Connection header received
0806 23:52:24.615502 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.615506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.615555 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.615622 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:24.615952 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.616660 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.618302 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.618753 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.618885 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.618944 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.619126 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.619205 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:52:24.620111 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.615013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:54987 (local address 127.10.225.131:46121)
0806 23:52:24.615150 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.615155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.615488 (+ 333us) server_negotiation.cc:407] Connection header received
0806 23:52:24.615566 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.615569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.615616 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.615718 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:24.616509 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.617236 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.618801 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.619321 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.619450 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.619499 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.619914 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.619993 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:52:24.620159 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.614912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:46121 (local address 127.10.225.130:54987)
0806 23:52:24.615371 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.615382 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.615486 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.615725 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.615728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.615753 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:24.616380 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.616391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.617436 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.617442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.618676 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.618686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.619606 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.619610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.619755 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.619784 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.619894 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.619999 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":37,"thread_start_us":152,"threads_started":1}
I20260806 23:52:24.620297 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d9b8d88e148daa773b168908600c1" candidate_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" is_pre_election: true
I20260806 23:52:24.620774 22933 raft_consensus.cc:2454] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec5831e2e0745a2b52a7e48ae1a9ae8 in term 0.
I20260806 23:52:24.620906 23303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d9b8d88e148daa773b168908600c1" candidate_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab8dac7f6de4764ac6c407661feccef" is_pre_election: true
I20260806 23:52:24.621408 23303 raft_consensus.cc:2454] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec5831e2e0745a2b52a7e48ae1a9ae8 in term 0.
I20260806 23:52:24.621502 23025 leader_election.cc:302] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec5831e2e0745a2b52a7e48ae1a9ae8, bb3670a2c69e4ffeadf61d88bf1ac51a; no voters:
I20260806 23:52:24.621841 23416 raft_consensus.cc:2781] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:24.621955 23416 raft_consensus.cc:482] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:24.622031 23416 raft_consensus.cc:3037] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:24.623149 23416 raft_consensus.cc:504] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } }
I20260806 23:52:24.623519 23416 leader_election.cc:288] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [CANDIDATE]: Term 1 election: Requested vote from peers aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121), bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319)
I20260806 23:52:24.623934 23303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d9b8d88e148daa773b168908600c1" candidate_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab8dac7f6de4764ac6c407661feccef"
I20260806 23:52:24.624035 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d9b8d88e148daa773b168908600c1" candidate_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a"
I20260806 23:52:24.624078 23303 raft_consensus.cc:3037] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:24.624194 22933 raft_consensus.cc:3037] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:24.625170 23303 raft_consensus.cc:2454] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec5831e2e0745a2b52a7e48ae1a9ae8 in term 1.
I20260806 23:52:24.625232 22933 raft_consensus.cc:2454] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec5831e2e0745a2b52a7e48ae1a9ae8 in term 1.
I20260806 23:52:24.625550 23029 leader_election.cc:302] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7ec5831e2e0745a2b52a7e48ae1a9ae8, aab8dac7f6de4764ac6c407661feccef, bb3670a2c69e4ffeadf61d88bf1ac51a; no voters:
I20260806 23:52:24.625772 23416 raft_consensus.cc:2781] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:24.626005 23416 raft_consensus.cc:686] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec5831e2e0745a2b52a7e48ae1a9ae8, State: Running, Role: LEADER
I20260806 23:52:24.626392 23416 consensus_queue.cc:237] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } }
I20260806 23:52:24.630511 22714 catalog_manager.cc:5495] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130). New cstate: current_term: 1 leader_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } health_report { overall_health: UNKNOWN } } }
I20260806 23:52:24.654958 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.651508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.10.225.130:35841)
0806 23:52:24.651661 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.651665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.651699 (+ 34us) server_negotiation.cc:407] Connection header received
0806 23:52:24.651777 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.651779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.651861 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.651954 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:24.652215 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.652722 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.653546 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.653758 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.653852 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.654111 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:24.654548 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:24.654649 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.654723 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.654787 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 23:52:24.660884 23303 raft_consensus.cc:1261] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Refusing update from remote peer 7ec5831e2e0745a2b52a7e48ae1a9ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:24.660884 22933 raft_consensus.cc:1261] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Refusing update from remote peer 7ec5831e2e0745a2b52a7e48ae1a9ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:24.661578 23420 consensus_queue.cc:1035] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:24.661758 23416 consensus_queue.cc:1035] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:24.671751 23425 mvcc.cc:205] Tried to move back new op lower bound from 7315703171723968512 to 7315703171592327168. Current Snapshot: MvccSnapshot[applied={T|T < 7315703171723968512}]
I20260806 23:52:24.674217 23427 mvcc.cc:205] Tried to move back new op lower bound from 7315703171723968512 to 7315703171592327168. Current Snapshot: MvccSnapshot[applied={T|T < 7315703171723968512}]
I20260806 23:52:24.678452 22713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:24.678644 22713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:24.681262 22713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: Sending DeleteTablet for 3 replicas of tablet 6e3d9b8d88e148daa773b168908600c1
I20260806 23:52:24.682215 23275 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e3d9b8d88e148daa773b168908600c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:24 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:46596
I20260806 23:52:24.682403 22903 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e3d9b8d88e148daa773b168908600c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:24 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:53458
I20260806 23:52:24.682723 23435 tablet_replica.cc:330] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: stopping tablet replica
I20260806 23:52:24.683060 23435 raft_consensus.cc:2229] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:24.683099 23091 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e3d9b8d88e148daa773b168908600c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:24 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:54088
I20260806 23:52:24.683334 23436 tablet_replica.cc:330] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: stopping tablet replica
I20260806 23:52:24.683399 23435 raft_consensus.cc:2258] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:24.683588 22713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 23:52:24.683714 23436 raft_consensus.cc:2229] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 23:52:24.684085 22713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:24.684271 23436 raft_consensus.cc:2258] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:24.685137 23435 ts_tablet_manager.cc:1905] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:24.685712 23436 ts_tablet_manager.cc:1905] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:24.688035 23435 ts_tablet_manager.cc:1918] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:24.688145 23435 log.cc:1196] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/6e3d9b8d88e148daa773b168908600c1
I20260806 23:52:24.688493 23435 ts_tablet_manager.cc:1939] T 6e3d9b8d88e148daa773b168908600c1 P aab8dac7f6de4764ac6c407661feccef: Deleting consensus metadata
I20260806 23:52:24.688652 23436 ts_tablet_manager.cc:1918] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:24.688804 23436 log.cc:1196] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/6e3d9b8d88e148daa773b168908600c1
I20260806 23:52:24.689144 23436 ts_tablet_manager.cc:1939] T 6e3d9b8d88e148daa773b168908600c1 P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting consensus metadata
I20260806 23:52:24.689925 23275 tablet_service.cc:1467] Processing CreateTablet for tablet a9f8d77493194a4cbaab74f70ed8b3fe (DEFAULT_TABLE table=test-table [id=a42128395595492f8d9a5b9dfcaf0048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:24.690199 23275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f8d77493194a4cbaab74f70ed8b3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:24.690264 22903 tablet_service.cc:1467] Processing CreateTablet for tablet a9f8d77493194a4cbaab74f70ed8b3fe (DEFAULT_TABLE table=test-table [id=a42128395595492f8d9a5b9dfcaf0048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:24.690622 22695 catalog_manager.cc:4841] TS bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319): tablet 6e3d9b8d88e148daa773b168908600c1 (table test-table [id=82850a0c08b840eca6285a0bdbacb1d2]) successfully deleted
I20260806 23:52:24.690682 22903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f8d77493194a4cbaab74f70ed8b3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:24.692258 23405 tablet_bootstrap.cc:492] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: Bootstrap starting.
I20260806 23:52:24.693073 23405 tablet_bootstrap.cc:654] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:24.693524 23407 tablet_bootstrap.cc:492] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: Bootstrap starting.
I20260806 23:52:24.693610 22698 catalog_manager.cc:4841] TS aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121): tablet 6e3d9b8d88e148daa773b168908600c1 (table test-table [id=82850a0c08b840eca6285a0bdbacb1d2]) successfully deleted
I20260806 23:52:24.694487 23407 tablet_bootstrap.cc:654] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:24.694456 23091 tablet_service.cc:1467] Processing CreateTablet for tablet a9f8d77493194a4cbaab74f70ed8b3fe (DEFAULT_TABLE table=test-table [id=a42128395595492f8d9a5b9dfcaf0048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:24.694741 23405 tablet_bootstrap.cc:492] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: No bootstrap required, opened a new log
I20260806 23:52:24.694793 23091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f8d77493194a4cbaab74f70ed8b3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:24.694820 23405 ts_tablet_manager.cc:1397] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 23:52:24.695279 23405 raft_consensus.cc:348] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.695405 23405 raft_consensus.cc:374] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:24.695438 23405 raft_consensus.cc:729] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab8dac7f6de4764ac6c407661feccef, State: Initialized, Role: FOLLOWER
I20260806 23:52:24.695564 23405 consensus_queue.cc:260] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.695665 23407 tablet_bootstrap.cc:492] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: No bootstrap required, opened a new log
I20260806 23:52:24.695775 23407 ts_tablet_manager.cc:1397] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 23:52:24.696368 23405 ts_tablet_manager.cc:1428] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 23:52:24.696763 23407 raft_consensus.cc:348] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.696892 23407 raft_consensus.cc:374] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:24.696923 23407 raft_consensus.cc:729] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3670a2c69e4ffeadf61d88bf1ac51a, State: Initialized, Role: FOLLOWER
I20260806 23:52:24.697045 23407 consensus_queue.cc:260] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.697324 23407 ts_tablet_manager.cc:1428] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 23:52:24.698083 23408 tablet_bootstrap.cc:492] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Bootstrap starting.
I20260806 23:52:24.699108 23408 tablet_bootstrap.cc:654] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:24.699867 23438 tablet_replica.cc:330] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: stopping tablet replica
I20260806 23:52:24.700232 23438 raft_consensus.cc:2229] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:52:24.700623 23438 raft_consensus.cc:2258] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:24.701004 23408 tablet_bootstrap.cc:492] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: No bootstrap required, opened a new log
I20260806 23:52:24.701092 23408 ts_tablet_manager.cc:1397] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:24.701561 23408 raft_consensus.cc:348] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.701681 23408 raft_consensus.cc:374] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:24.701717 23408 raft_consensus.cc:729] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec5831e2e0745a2b52a7e48ae1a9ae8, State: Initialized, Role: FOLLOWER
I20260806 23:52:24.701855 23408 consensus_queue.cc:260] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.702131 23408 ts_tablet_manager.cc:1428] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:52:24.702946 23438 ts_tablet_manager.cc:1905] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:24.706022 23438 ts_tablet_manager.cc:1918] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:24.706157 23438 log.cc:1196] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/6e3d9b8d88e148daa773b168908600c1
I20260806 23:52:24.706486 23438 ts_tablet_manager.cc:1939] T 6e3d9b8d88e148daa773b168908600c1 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting consensus metadata
I20260806 23:52:24.708091 22695 catalog_manager.cc:4841] TS 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841): tablet 6e3d9b8d88e148daa773b168908600c1 (table test-table [id=82850a0c08b840eca6285a0bdbacb1d2]) successfully deleted
W20260806 23:52:24.728084 23371 tablet.cc:2369] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:52:24.741075 23182 tablet.cc:2369] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:52:24.748039 22993 tablet.cc:2369] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:52:24.764542 23414 raft_consensus.cc:482] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:24.764730 23414 raft_consensus.cc:504] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.766073 23414 leader_election.cc:288] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319), 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841)
I20260806 23:52:24.770092 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.766277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:46319 (local address 127.10.225.131:46971)
0806 23:52:24.766410 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.766426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.766512 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.766914 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.766918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.766939 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:24.767065 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.767072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.768181 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.768184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.768993 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.769003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.769651 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.769655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.769772 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.769791 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.769854 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.769933 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260806 23:52:24.770076 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.766481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:46971 (local address 127.10.225.129:46319)
0806 23:52:24.766602 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.766606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.766621 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:52:24.766677 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.766680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.766724 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.766796 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:24.767226 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.768069 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.769116 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.769544 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.769651 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.769709 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.769880 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.769956 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:52:24.770586 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f8d77493194a4cbaab74f70ed8b3fe" candidate_uuid: "aab8dac7f6de4764ac6c407661feccef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" is_pre_election: true
I20260806 23:52:24.770737 22933 raft_consensus.cc:2454] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aab8dac7f6de4764ac6c407661feccef in term 0.
I20260806 23:52:24.770993 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.766550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35841 (local address 127.10.225.131:57103)
0806 23:52:24.766856 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.766869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.767003 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.767357 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.767359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.767374 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:24.767472 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.767477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.768227 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.768231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.769574 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.769586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.770596 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.770600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.770697 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.770719 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.770786 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.770851 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260806 23:52:24.771018 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.766767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:57103 (local address 127.10.225.130:35841)
0806 23:52:24.766881 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.766886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.766938 (+ 52us) server_negotiation.cc:407] Connection header received
0806 23:52:24.767128 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.767132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.767185 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.767234 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:24.767584 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.768119 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.769729 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.770336 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.770431 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.770497 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.770815 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.770898 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:52:24.771133 23210 leader_election.cc:302] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aab8dac7f6de4764ac6c407661feccef, bb3670a2c69e4ffeadf61d88bf1ac51a; no voters:
I20260806 23:52:24.771431 23414 raft_consensus.cc:2781] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:24.771502 23414 raft_consensus.cc:482] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:24.771535 23414 raft_consensus.cc:3037] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:24.771832 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f8d77493194a4cbaab74f70ed8b3fe" candidate_uuid: "aab8dac7f6de4764ac6c407661feccef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" is_pre_election: true
I20260806 23:52:24.772050 23117 raft_consensus.cc:2454] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aab8dac7f6de4764ac6c407661feccef in term 0.
I20260806 23:52:24.772898 23414 raft_consensus.cc:504] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.773278 23414 leader_election.cc:288] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [CANDIDATE]: Term 1 election: Requested vote from peers bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319), 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841)
I20260806 23:52:24.773990 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f8d77493194a4cbaab74f70ed8b3fe" candidate_uuid: "aab8dac7f6de4764ac6c407661feccef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a"
I20260806 23:52:24.774129 22933 raft_consensus.cc:3037] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:24.774307 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f8d77493194a4cbaab74f70ed8b3fe" candidate_uuid: "aab8dac7f6de4764ac6c407661feccef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8"
I20260806 23:52:24.774425 23117 raft_consensus.cc:3037] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:24.775389 22933 raft_consensus.cc:2454] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aab8dac7f6de4764ac6c407661feccef in term 1.
I20260806 23:52:24.775452 23117 raft_consensus.cc:2454] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aab8dac7f6de4764ac6c407661feccef in term 1.
I20260806 23:52:24.775743 23210 leader_election.cc:302] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aab8dac7f6de4764ac6c407661feccef, bb3670a2c69e4ffeadf61d88bf1ac51a; no voters:
I20260806 23:52:24.775969 23414 raft_consensus.cc:2781] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:24.776211 23414 raft_consensus.cc:686] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 1 LEADER]: Becoming Leader. State: Replica: aab8dac7f6de4764ac6c407661feccef, State: Running, Role: LEADER
I20260806 23:52:24.776484 23414 consensus_queue.cc:237] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.779209 22713 catalog_manager.cc:5495] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef reported cstate change: term changed from 0 to 1, leader changed from <none> to aab8dac7f6de4764ac6c407661feccef (127.10.225.131). New cstate: current_term: 1 leader_uuid: "aab8dac7f6de4764ac6c407661feccef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } health_report { overall_health: HEALTHY } } }
Aug 06 23:52:24 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060343, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.190@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:24.817052 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.806109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.10.225.190:35377)
0806 23:52:24.806265 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.806270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.806508 (+ 238us) server_negotiation.cc:407] Connection header received
0806 23:52:24.806643 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.806645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.806691 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.806777 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:24.807737 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.808274 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.809469 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.809867 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.810036 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.813053 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:24.813057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:24.813059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:24.813086 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:24.814840 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.815401 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.815404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.815405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.815480 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:24.816013 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:24.816018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:24.816020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:24.816254 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:24.816376 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:24.816689 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 23:52:24.816923 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":57}
I20260806 23:52:24.817052 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.806019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35377 (local address 127.0.0.1:49558)
0806 23:52:24.806393 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.806414 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.806556 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.806814 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.806817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.807422 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:24.807571 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.807579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.808415 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.808419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.809316 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.809328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:24.810072 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:24.810077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:24.810173 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:24.810819 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:24.810835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:24.812848 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:24.814983 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.814997 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.815007 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.815293 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.815614 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:24.815631 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:24.815633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:24.815872 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:24.816385 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:24.816393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:24.816513 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 23:52:24.816622 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 23:52:24.816839 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260806 23:52:24.822046 22713 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:49558:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 23:52:24.823107 22713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:24.829707 22903 tablet_service.cc:1467] Processing CreateTablet for tablet 1abf1d1c899a4471bd0a8b0f643f0112 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ac26ddeef6d47f89a4e7500ccc2d59a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:52:24.829910 23091 tablet_service.cc:1467] Processing CreateTablet for tablet 1abf1d1c899a4471bd0a8b0f643f0112 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ac26ddeef6d47f89a4e7500ccc2d59a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:52:24.830276 23091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abf1d1c899a4471bd0a8b0f643f0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:24.830296 22903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abf1d1c899a4471bd0a8b0f643f0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:24.833388 23408 tablet_bootstrap.cc:492] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Bootstrap starting.
I20260806 23:52:24.833686 23275 tablet_service.cc:1467] Processing CreateTablet for tablet 1abf1d1c899a4471bd0a8b0f643f0112 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ac26ddeef6d47f89a4e7500ccc2d59a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 23:52:24.834036 23275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1abf1d1c899a4471bd0a8b0f643f0112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:24.834389 23408 tablet_bootstrap.cc:654] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:24.835748 23408 tablet_bootstrap.cc:492] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: No bootstrap required, opened a new log
I20260806 23:52:24.835840 23408 ts_tablet_manager.cc:1397] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:24.836299 23408 raft_consensus.cc:348] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.836448 23408 raft_consensus.cc:374] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:24.836480 23408 raft_consensus.cc:729] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec5831e2e0745a2b52a7e48ae1a9ae8, State: Initialized, Role: FOLLOWER
I20260806 23:52:24.836647 23408 consensus_queue.cc:260] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.836773 23405 tablet_bootstrap.cc:492] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: Bootstrap starting.
I20260806 23:52:24.836882 23416 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:24.837001 23416 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:52:24.837468 23407 tablet_bootstrap.cc:492] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: Bootstrap starting.
I20260806 23:52:24.837983 23408 ts_tablet_manager.cc:1428] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:52:24.838816 23405 tablet_bootstrap.cc:654] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:24.838814 23407 tablet_bootstrap.cc:654] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:24.840813 23407 tablet_bootstrap.cc:492] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: No bootstrap required, opened a new log
I20260806 23:52:24.840917 23407 ts_tablet_manager.cc:1397] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 23:52:24.841369 23407 raft_consensus.cc:348] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.841487 23407 raft_consensus.cc:374] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:24.841521 23407 raft_consensus.cc:729] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3670a2c69e4ffeadf61d88bf1ac51a, State: Initialized, Role: FOLLOWER
I20260806 23:52:24.841643 23407 consensus_queue.cc:260] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.841740 23405 tablet_bootstrap.cc:492] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: No bootstrap required, opened a new log
I20260806 23:52:24.841838 23405 ts_tablet_manager.cc:1397] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260806 23:52:24.841997 23407 ts_tablet_manager.cc:1428] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:24.842187 23418 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:24.842352 23418 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:52:24.842623 23405 raft_consensus.cc:348] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.842792 23405 raft_consensus.cc:374] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:24.842854 23405 raft_consensus.cc:729] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab8dac7f6de4764ac6c407661feccef, State: Initialized, Role: FOLLOWER
I20260806 23:52:24.842988 23405 consensus_queue.cc:260] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.843196 23414 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:24.843312 23414 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:52:24.843547 23405 ts_tablet_manager.cc:1428] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 23:52:24.963400 23416 raft_consensus.cc:482] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:24.963616 23416 raft_consensus.cc:504] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.964176 23416 leader_election.cc:288] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319), aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121)
I20260806 23:52:24.964820 23303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abf1d1c899a4471bd0a8b0f643f0112" candidate_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab8dac7f6de4764ac6c407661feccef" is_pre_election: true
I20260806 23:52:24.964875 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abf1d1c899a4471bd0a8b0f643f0112" candidate_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" is_pre_election: true
I20260806 23:52:24.965051 22933 raft_consensus.cc:2454] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec5831e2e0745a2b52a7e48ae1a9ae8 in term 0.
I20260806 23:52:24.965039 23303 raft_consensus.cc:2454] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec5831e2e0745a2b52a7e48ae1a9ae8 in term 0.
I20260806 23:52:24.965447 23025 leader_election.cc:302] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec5831e2e0745a2b52a7e48ae1a9ae8, bb3670a2c69e4ffeadf61d88bf1ac51a; no voters:
I20260806 23:52:24.965704 23416 raft_consensus.cc:2781] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:24.965770 23416 raft_consensus.cc:482] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:24.965817 23416 raft_consensus.cc:3037] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:24.966845 23416 raft_consensus.cc:504] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.967227 23416 leader_election.cc:288] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [CANDIDATE]: Term 1 election: Requested vote from peers bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319), aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121)
I20260806 23:52:24.967819 23303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abf1d1c899a4471bd0a8b0f643f0112" candidate_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab8dac7f6de4764ac6c407661feccef"
I20260806 23:52:24.967820 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1abf1d1c899a4471bd0a8b0f643f0112" candidate_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a"
I20260806 23:52:24.968035 22933 raft_consensus.cc:3037] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:24.968029 23303 raft_consensus.cc:3037] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:24.969484 22933 raft_consensus.cc:2454] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec5831e2e0745a2b52a7e48ae1a9ae8 in term 1.
I20260806 23:52:24.969520 23303 raft_consensus.cc:2454] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec5831e2e0745a2b52a7e48ae1a9ae8 in term 1.
I20260806 23:52:24.969928 23029 leader_election.cc:302] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7ec5831e2e0745a2b52a7e48ae1a9ae8, aab8dac7f6de4764ac6c407661feccef, bb3670a2c69e4ffeadf61d88bf1ac51a; no voters:
I20260806 23:52:24.970182 23416 raft_consensus.cc:2781] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:24.970288 23416 raft_consensus.cc:686] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec5831e2e0745a2b52a7e48ae1a9ae8, State: Running, Role: LEADER
I20260806 23:52:24.970463 23416 consensus_queue.cc:237] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:24.971653 23420 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: TxnStatusTablet state changed. Reason: New leader 7ec5831e2e0745a2b52a7e48ae1a9ae8. Latest consensus state: current_term: 1 leader_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:24.971853 23420 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:52:24.972237 23464 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 23:52:24.972435 23464 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 23:52:24.972795 22713 catalog_manager.cc:5495] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130). New cstate: current_term: 1 leader_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } health_report { overall_health: UNKNOWN } } }
I20260806 23:52:25.003947 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.998947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.10.225.130:35841)
0806 23:52:24.999454 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:24.999459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:24.999473 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:24.999525 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:24.999529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:24.999584 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:24.999658 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:25.000156 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.000953 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.002423 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.002841 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.002959 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.003326 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.003427 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.003501 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.003701 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.003765 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260806 23:52:25.004012 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:24.998638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35841 (local address 127.0.0.1:50294)
0806 23:52:24.999137 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:24.999153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:24.999270 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:24.999788 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:24.999792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:24.999808 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:52:24.999979 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:24.999996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.001096 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.001100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.002252 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.002266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.002973 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.002977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.003087 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.003170 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.003511 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.003524 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 23:52:25.003612 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 23:52:25.003772 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369}
I20260806 23:52:25.007947 23303 raft_consensus.cc:1261] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Refusing update from remote peer 7ec5831e2e0745a2b52a7e48ae1a9ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:25.007947 22933 raft_consensus.cc:1261] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Refusing update from remote peer 7ec5831e2e0745a2b52a7e48ae1a9ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:25.008608 23416 consensus_queue.cc:1035] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:25.008924 23420 consensus_queue.cc:1035] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:25.010499 23418 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: TxnStatusTablet state changed. Reason: New leader 7ec5831e2e0745a2b52a7e48ae1a9ae8. Latest consensus state: current_term: 1 leader_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:25.010499 23414 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: TxnStatusTablet state changed. Reason: New leader 7ec5831e2e0745a2b52a7e48ae1a9ae8. Latest consensus state: current_term: 1 leader_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:25.010624 23414 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:52:25.010624 23418 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:52:25.014072 23420 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:25.014261 23420 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:52:25.014370 23424 mvcc.cc:205] Tried to move back new op lower bound from 7315703173146542080 to 7315703173000708096. Current Snapshot: MvccSnapshot[applied={T|T < 7315703173146542080 or (T in {7315703173146542080})}]
I20260806 23:52:25.016713 23418 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:25.016855 23418 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P bb3670a2c69e4ffeadf61d88bf1ac51a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:52:25.018967 23420 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:25.019205 23420 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: This TxnStatusTablet replica's current role is: LEADER
I20260806 23:52:25.021283 23414 tablet_replica.cc:439] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } } }
I20260806 23:52:25.021442 23414 tablet_replica.cc:442] T 1abf1d1c899a4471bd0a8b0f643f0112 P aab8dac7f6de4764ac6c407661feccef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 23:52:25.025219 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.019762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.10.225.131:46121)
0806 23:52:25.019891 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.019896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.019912 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:52:25.020001 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.020003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.020073 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.020229 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:25.021923 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.022763 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.023644 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.023952 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.024046 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.024332 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.024815 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.024931 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.025004 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.025065 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:52:25.038103 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.033997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35841 (local address 127.10.225.131:43437)
0806 23:52:25.034400 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:25.034412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:25.034511 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:25.034836 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:25.034840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:25.034854 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:52:25.035040 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.035048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.036148 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.036151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.036811 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.036821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.037391 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.037394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.037478 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.037534 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.037826 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.037830 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:52:25.037883 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 23:52:25.037966 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":134,"threads_started":1}
I20260806 23:52:25.038105 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.034128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:43437 (local address 127.10.225.130:35841)
0806 23:52:25.034266 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.034271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.034476 (+ 205us) server_negotiation.cc:407] Connection header received
0806 23:52:25.034599 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.034602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.034659 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.034742 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:25.035190 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.036016 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.036920 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.037255 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.037353 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.037662 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.037751 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.037820 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.037910 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.037965 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:52:25.045670 22933 raft_consensus.cc:1261] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Refusing update from remote peer aab8dac7f6de4764ac6c407661feccef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:25.045992 23117 raft_consensus.cc:1261] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Refusing update from remote peer aab8dac7f6de4764ac6c407661feccef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:25.046196 23414 consensus_queue.cc:1035] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:25.046568 23414 consensus_queue.cc:1035] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:25.070313 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.066925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:32959 (local address 127.10.225.131:46121)
0806 23:52:25.067060 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.067065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.067080 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:52:25.067126 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.067128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.067181 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.067261 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:25.067636 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.068209 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.069061 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.069352 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.069437 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.069864 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.069942 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.069977 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.070111 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.070170 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:52:25.070818 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.066445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:46121 (local address 127.10.225.130:32959)
0806 23:52:25.066802 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:25.066818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:25.066912 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:25.067400 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:25.067403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:25.067412 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:52:25.067525 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.067530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.068314 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.068318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.068931 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.068940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.069568 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.069572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.069659 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.069714 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.070024 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.070028 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:52:25.070613 (+ 585us) client_negotiation.cc:240] Negotiation successful
0806 23:52:25.070706 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":131,"threads_started":1}
I20260806 23:52:25.098574 22716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:25.098774 22716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:25.101394 22716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: Sending DeleteTablet for 3 replicas of tablet a9f8d77493194a4cbaab74f70ed8b3fe
I20260806 23:52:25.102141 22903 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f8d77493194a4cbaab74f70ed8b3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:25 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:53458
I20260806 23:52:25.102214 23091 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f8d77493194a4cbaab74f70ed8b3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:25 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:54088
I20260806 23:52:25.102398 23436 tablet_replica.cc:330] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: stopping tablet replica
I20260806 23:52:25.102450 23438 tablet_replica.cc:330] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: stopping tablet replica
I20260806 23:52:25.102532 23436 raft_consensus.cc:2229] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:25.102578 23438 raft_consensus.cc:2229] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:25.102665 23436 raft_consensus.cc:2258] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:25.102700 23438 raft_consensus.cc:2258] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:25.103497 23275 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f8d77493194a4cbaab74f70ed8b3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:25 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:46596
I20260806 23:52:25.103813 23435 tablet_replica.cc:330] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: stopping tablet replica
I20260806 23:52:25.103948 23435 raft_consensus.cc:2229] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:52:25.104355 23435 raft_consensus.cc:2258] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:25.105420 23435 ts_tablet_manager.cc:1905] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:25.108811 23438 ts_tablet_manager.cc:1905] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:25.108829 23435 ts_tablet_manager.cc:1918] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:52:25.108909 23435 log.cc:1196] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/a9f8d77493194a4cbaab74f70ed8b3fe
I20260806 23:52:25.109172 23435 ts_tablet_manager.cc:1939] T a9f8d77493194a4cbaab74f70ed8b3fe P aab8dac7f6de4764ac6c407661feccef: Deleting consensus metadata
I20260806 23:52:25.111835 23438 ts_tablet_manager.cc:1918] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:52:25.111928 23438 log.cc:1196] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/a9f8d77493194a4cbaab74f70ed8b3fe
I20260806 23:52:25.112304 22698 catalog_manager.cc:4841] TS aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121): tablet a9f8d77493194a4cbaab74f70ed8b3fe (table test-table [id=a42128395595492f8d9a5b9dfcaf0048]) successfully deleted
I20260806 23:52:25.112991 23436 ts_tablet_manager.cc:1905] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:25.115792 23438 ts_tablet_manager.cc:1939] T a9f8d77493194a4cbaab74f70ed8b3fe P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting consensus metadata
I20260806 23:52:25.116046 23436 ts_tablet_manager.cc:1918] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:52:25.116132 23436 log.cc:1196] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/a9f8d77493194a4cbaab74f70ed8b3fe
I20260806 23:52:25.116387 23436 ts_tablet_manager.cc:1939] T a9f8d77493194a4cbaab74f70ed8b3fe P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting consensus metadata
I20260806 23:52:25.116931 22695 catalog_manager.cc:4841] TS 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841): tablet a9f8d77493194a4cbaab74f70ed8b3fe (table test-table [id=a42128395595492f8d9a5b9dfcaf0048]) successfully deleted
I20260806 23:52:25.117635 22695 catalog_manager.cc:4841] TS bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319): tablet a9f8d77493194a4cbaab74f70ed8b3fe (table test-table [id=a42128395595492f8d9a5b9dfcaf0048]) successfully deleted
Aug 06 23:52:25 dist-test-slave-nj21 krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 23:52:25 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:25 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 23:52:25.146433 22780 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:25.140132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.10.225.190:35377)
0806 23:52:25.140293 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.140309 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.140416 (+ 107us) server_negotiation.cc:407] Connection header received
0806 23:52:25.140536 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.140539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.140590 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.140690 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:25.141431 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.141957 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.142784 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.143001 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.143089 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.146295 (+ 3206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49568: BlockingRecv error: recv got EOF from 127.0.0.1:49568 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 23:52:25 dist-test-slave-nj21 krb5kdc[22653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM
I20260806 23:52:25.157869 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.148226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.10.225.190:35377)
0806 23:52:25.148353 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.148356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.148367 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:52:25.148400 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.148402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.148447 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.148510 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:25.149073 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.149594 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.150351 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.150569 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.150655 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.153839 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:25.153844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:25.153846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:25.153891 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:25.155774 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:25.156483 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:25.156487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:25.156489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:25.156602 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:25.156944 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:25.156948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:25.156951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:25.157209 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:25.157337 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.157588 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.157717 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:52:25.160758 22716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:52:25.161322 22716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:25.166826 22903 tablet_service.cc:1467] Processing CreateTablet for tablet e617a5e082a548d4a2360b58aa84c845 (DEFAULT_TABLE table=test-table [id=5884150838ad4fc99fca85a6c2c6db39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:25.166991 23091 tablet_service.cc:1467] Processing CreateTablet for tablet e617a5e082a548d4a2360b58aa84c845 (DEFAULT_TABLE table=test-table [id=5884150838ad4fc99fca85a6c2c6db39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:25.167193 22903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e617a5e082a548d4a2360b58aa84c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:25.167096 23275 tablet_service.cc:1467] Processing CreateTablet for tablet e617a5e082a548d4a2360b58aa84c845 (DEFAULT_TABLE table=test-table [id=5884150838ad4fc99fca85a6c2c6db39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:25.167249 23091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e617a5e082a548d4a2360b58aa84c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:25.167536 23275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e617a5e082a548d4a2360b58aa84c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:25.169806 23408 tablet_bootstrap.cc:492] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Bootstrap starting.
I20260806 23:52:25.170543 23405 tablet_bootstrap.cc:492] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: Bootstrap starting.
I20260806 23:52:25.170696 23408 tablet_bootstrap.cc:654] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:25.171496 23407 tablet_bootstrap.cc:492] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: Bootstrap starting.
I20260806 23:52:25.171602 23405 tablet_bootstrap.cc:654] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:25.172551 23407 tablet_bootstrap.cc:654] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:25.173810 23407 tablet_bootstrap.cc:492] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: No bootstrap required, opened a new log
I20260806 23:52:25.173818 23408 tablet_bootstrap.cc:492] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: No bootstrap required, opened a new log
I20260806 23:52:25.173903 23408 ts_tablet_manager.cc:1397] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 23:52:25.173903 23407 ts_tablet_manager.cc:1397] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 23:52:25.174356 23407 raft_consensus.cc:348] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.174356 23408 raft_consensus.cc:348] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.174484 23407 raft_consensus.cc:374] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:25.174484 23408 raft_consensus.cc:374] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:25.174517 23407 raft_consensus.cc:729] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3670a2c69e4ffeadf61d88bf1ac51a, State: Initialized, Role: FOLLOWER
I20260806 23:52:25.174517 23408 raft_consensus.cc:729] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec5831e2e0745a2b52a7e48ae1a9ae8, State: Initialized, Role: FOLLOWER
I20260806 23:52:25.174649 23408 consensus_queue.cc:260] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.174649 23407 consensus_queue.cc:260] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.174916 23408 ts_tablet_manager.cc:1428] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:25.174914 23407 ts_tablet_manager.cc:1428] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:52:25.177918 23405 tablet_bootstrap.cc:492] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: No bootstrap required, opened a new log
I20260806 23:52:25.178032 23405 ts_tablet_manager.cc:1397] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260806 23:52:25.178498 23405 raft_consensus.cc:348] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.178608 23405 raft_consensus.cc:374] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:25.178628 23405 raft_consensus.cc:729] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab8dac7f6de4764ac6c407661feccef, State: Initialized, Role: FOLLOWER
I20260806 23:52:25.178741 23405 consensus_queue.cc:260] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.179016 23405 ts_tablet_manager.cc:1428] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 23:52:25.212294 23414 raft_consensus.cc:482] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:25.212594 23414 raft_consensus.cc:504] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.213250 23414 leader_election.cc:288] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319), 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841)
I20260806 23:52:25.213788 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617a5e082a548d4a2360b58aa84c845" candidate_uuid: "aab8dac7f6de4764ac6c407661feccef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" is_pre_election: true
I20260806 23:52:25.213788 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617a5e082a548d4a2360b58aa84c845" candidate_uuid: "aab8dac7f6de4764ac6c407661feccef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" is_pre_election: true
I20260806 23:52:25.213984 22933 raft_consensus.cc:2454] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aab8dac7f6de4764ac6c407661feccef in term 0.
I20260806 23:52:25.213984 23117 raft_consensus.cc:2454] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aab8dac7f6de4764ac6c407661feccef in term 0.
I20260806 23:52:25.214359 23210 leader_election.cc:302] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aab8dac7f6de4764ac6c407661feccef, bb3670a2c69e4ffeadf61d88bf1ac51a; no voters:
I20260806 23:52:25.214576 23414 raft_consensus.cc:2781] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:25.214636 23414 raft_consensus.cc:482] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:25.214674 23414 raft_consensus.cc:3037] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:25.216058 23414 raft_consensus.cc:504] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.216470 23414 leader_election.cc:288] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [CANDIDATE]: Term 1 election: Requested vote from peers bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319), 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841)
I20260806 23:52:25.216933 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617a5e082a548d4a2360b58aa84c845" candidate_uuid: "aab8dac7f6de4764ac6c407661feccef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a"
I20260806 23:52:25.217003 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e617a5e082a548d4a2360b58aa84c845" candidate_uuid: "aab8dac7f6de4764ac6c407661feccef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8"
I20260806 23:52:25.217110 22933 raft_consensus.cc:3037] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:25.217167 23117 raft_consensus.cc:3037] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:25.218582 23117 raft_consensus.cc:2454] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aab8dac7f6de4764ac6c407661feccef in term 1.
I20260806 23:52:25.218582 22933 raft_consensus.cc:2454] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aab8dac7f6de4764ac6c407661feccef in term 1.
I20260806 23:52:25.218986 23210 leader_election.cc:302] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec5831e2e0745a2b52a7e48ae1a9ae8, aab8dac7f6de4764ac6c407661feccef; no voters:
I20260806 23:52:25.219180 23414 raft_consensus.cc:2781] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:25.219318 23414 raft_consensus.cc:686] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 1 LEADER]: Becoming Leader. State: Replica: aab8dac7f6de4764ac6c407661feccef, State: Running, Role: LEADER
I20260806 23:52:25.219506 23414 consensus_queue.cc:237] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.221928 22713 catalog_manager.cc:5495] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef reported cstate change: term changed from 0 to 1, leader changed from <none> to aab8dac7f6de4764ac6c407661feccef (127.10.225.131). New cstate: current_term: 1 leader_uuid: "aab8dac7f6de4764ac6c407661feccef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } health_report { overall_health: HEALTHY } } }
I20260806 23:52:25.233659 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.229501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.10.225.131:46121)
0806 23:52:25.229703 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.229708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.229932 (+ 224us) server_negotiation.cc:407] Connection header received
0806 23:52:25.230075 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.230077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.230131 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.230199 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:25.230507 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.231005 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.232286 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.232574 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.232703 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.233053 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.233155 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.233221 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.233417 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.233500 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 23:52:25.236253 23117 raft_consensus.cc:1261] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Refusing update from remote peer aab8dac7f6de4764ac6c407661feccef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:25.236759 22933 raft_consensus.cc:1261] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Refusing update from remote peer aab8dac7f6de4764ac6c407661feccef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:25.237450 23414 consensus_queue.cc:1035] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:25.237696 23445 consensus_queue.cc:1035] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:25.253345 22713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:25.253557 22713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:25.256814 22713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: Sending DeleteTablet for 3 replicas of tablet e617a5e082a548d4a2360b58aa84c845
I20260806 23:52:25.257558 22903 tablet_service.cc:1514] Processing DeleteTablet for tablet e617a5e082a548d4a2360b58aa84c845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:25 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:53458
I20260806 23:52:25.257583 23275 tablet_service.cc:1514] Processing DeleteTablet for tablet e617a5e082a548d4a2360b58aa84c845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:25 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:46596
I20260806 23:52:25.257773 23435 tablet_replica.cc:330] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: stopping tablet replica
I20260806 23:52:25.257751 23091 tablet_service.cc:1514] Processing DeleteTablet for tablet e617a5e082a548d4a2360b58aa84c845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:25 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:54088
I20260806 23:52:25.257890 23435 raft_consensus.cc:2229] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:52:25.257922 23438 tablet_replica.cc:330] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: stopping tablet replica
I20260806 23:52:25.258035 23438 raft_consensus.cc:2229] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:25.258095 23435 raft_consensus.cc:2258] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:25.258169 23438 raft_consensus.cc:2258] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:25.258271 23436 tablet_replica.cc:330] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: stopping tablet replica
I20260806 23:52:25.258386 23436 raft_consensus.cc:2229] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:25.258510 23436 raft_consensus.cc:2258] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:25.259171 23436 ts_tablet_manager.cc:1905] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:25.259353 23435 ts_tablet_manager.cc:1905] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:25.261758 23436 ts_tablet_manager.cc:1918] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:25.261826 23436 log.cc:1196] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/e617a5e082a548d4a2360b58aa84c845
I20260806 23:52:25.261943 23435 ts_tablet_manager.cc:1918] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:25.262006 23435 log.cc:1196] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/e617a5e082a548d4a2360b58aa84c845
I20260806 23:52:25.262053 23436 ts_tablet_manager.cc:1939] T e617a5e082a548d4a2360b58aa84c845 P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting consensus metadata
I20260806 23:52:25.262235 23435 ts_tablet_manager.cc:1939] T e617a5e082a548d4a2360b58aa84c845 P aab8dac7f6de4764ac6c407661feccef: Deleting consensus metadata
I20260806 23:52:25.262796 22695 catalog_manager.cc:4841] TS bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319): tablet e617a5e082a548d4a2360b58aa84c845 (table test-table [id=5884150838ad4fc99fca85a6c2c6db39]) successfully deleted
I20260806 23:52:25.262933 22698 catalog_manager.cc:4841] TS aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121): tablet e617a5e082a548d4a2360b58aa84c845 (table test-table [id=5884150838ad4fc99fca85a6c2c6db39]) successfully deleted
I20260806 23:52:25.264309 23438 ts_tablet_manager.cc:1905] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:25.264267 22713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:52:25.264752 22713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:25.266614 23438 ts_tablet_manager.cc:1918] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:25.266702 23438 log.cc:1196] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/e617a5e082a548d4a2360b58aa84c845
I20260806 23:52:25.266963 23438 ts_tablet_manager.cc:1939] T e617a5e082a548d4a2360b58aa84c845 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting consensus metadata
I20260806 23:52:25.268560 22695 catalog_manager.cc:4841] TS 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841): tablet e617a5e082a548d4a2360b58aa84c845 (table test-table [id=5884150838ad4fc99fca85a6c2c6db39]) successfully deleted
I20260806 23:52:25.270009 22903 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5e40e7ccba46b49f3f3a4f97fbca08 (DEFAULT_TABLE table=test-table [id=8690cd57a91b497e87f38903f956b17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:25.270345 22903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5e40e7ccba46b49f3f3a4f97fbca08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:25.273099 23407 tablet_bootstrap.cc:492] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: Bootstrap starting.
I20260806 23:52:25.274144 23407 tablet_bootstrap.cc:654] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:25.275460 23407 tablet_bootstrap.cc:492] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: No bootstrap required, opened a new log
I20260806 23:52:25.275534 23407 ts_tablet_manager.cc:1397] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 23:52:25.275933 23275 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5e40e7ccba46b49f3f3a4f97fbca08 (DEFAULT_TABLE table=test-table [id=8690cd57a91b497e87f38903f956b17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:25.276115 23407 raft_consensus.cc:348] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.276213 23407 raft_consensus.cc:374] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:25.276234 23407 raft_consensus.cc:729] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3670a2c69e4ffeadf61d88bf1ac51a, State: Initialized, Role: FOLLOWER
I20260806 23:52:25.276238 23275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5e40e7ccba46b49f3f3a4f97fbca08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:25.276341 23407 consensus_queue.cc:260] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.276584 23407 ts_tablet_manager.cc:1428] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:25.278683 23405 tablet_bootstrap.cc:492] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: Bootstrap starting.
I20260806 23:52:25.279220 23091 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5e40e7ccba46b49f3f3a4f97fbca08 (DEFAULT_TABLE table=test-table [id=8690cd57a91b497e87f38903f956b17d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:25.279503 23091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5e40e7ccba46b49f3f3a4f97fbca08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:25.279587 23405 tablet_bootstrap.cc:654] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:25.280663 23405 tablet_bootstrap.cc:492] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: No bootstrap required, opened a new log
I20260806 23:52:25.280751 23405 ts_tablet_manager.cc:1397] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:52:25.281210 23405 raft_consensus.cc:348] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.281325 23405 raft_consensus.cc:374] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:25.281356 23405 raft_consensus.cc:729] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab8dac7f6de4764ac6c407661feccef, State: Initialized, Role: FOLLOWER
I20260806 23:52:25.281486 23405 consensus_queue.cc:260] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.281759 23405 ts_tablet_manager.cc:1428] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:25.282083 23408 tablet_bootstrap.cc:492] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Bootstrap starting.
I20260806 23:52:25.282941 23408 tablet_bootstrap.cc:654] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:25.284107 23408 tablet_bootstrap.cc:492] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: No bootstrap required, opened a new log
I20260806 23:52:25.284209 23408 ts_tablet_manager.cc:1397] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:52:25.284621 23408 raft_consensus.cc:348] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.284739 23408 raft_consensus.cc:374] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:25.284771 23408 raft_consensus.cc:729] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec5831e2e0745a2b52a7e48ae1a9ae8, State: Initialized, Role: FOLLOWER
I20260806 23:52:25.284909 23408 consensus_queue.cc:260] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.285187 23408 ts_tablet_manager.cc:1428] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:52:25.382928 23418 raft_consensus.cc:482] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:25.383090 23418 raft_consensus.cc:504] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.384310 23418 leader_election.cc:288] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841), aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121)
I20260806 23:52:25.388155 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.384789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:36953 (local address 127.10.225.131:46121)
0806 23:52:25.384914 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.384919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.385039 (+ 120us) server_negotiation.cc:407] Connection header received
0806 23:52:25.385206 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.385209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.385272 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.385335 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:25.385685 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.386181 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.387227 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.387623 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.387756 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.387819 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.387966 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.388034 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 23:52:25.388185 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.384601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:46121 (local address 127.10.225.129:36953)
0806 23:52:25.384952 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:25.384966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:25.385109 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:25.385369 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:25.385372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:25.385388 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:25.385571 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.385579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.386285 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.386288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.387112 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.387121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.387756 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.387760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.387851 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.387874 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:52:25.387941 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 23:52:25.388011 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":42,"spinlock_wait_cycles":61696,"thread_start_us":101,"threads_started":1}
I20260806 23:52:25.388700 23303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5e40e7ccba46b49f3f3a4f97fbca08" candidate_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab8dac7f6de4764ac6c407661feccef" is_pre_election: true
I20260806 23:52:25.388790 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.384718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:52845 (local address 127.10.225.130:35841)
0806 23:52:25.384859 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.384866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.385040 (+ 174us) server_negotiation.cc:407] Connection header received
0806 23:52:25.385193 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.385195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.385237 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.385308 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:25.385592 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.386111 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.387250 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.387982 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.388186 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.388254 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.388543 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.388634 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:52:25.388811 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.384616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35841 (local address 127.10.225.129:52845)
0806 23:52:25.384957 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:25.384969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:25.385090 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:25.385321 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:25.385323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:25.385339 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:25.385488 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.385496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.386230 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.386234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.387125 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.387135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.388208 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.388213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.388368 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.388404 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 23:52:25.388514 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 23:52:25.388621 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":33,"thread_start_us":83,"threads_started":1}
I20260806 23:52:25.388859 23303 raft_consensus.cc:2454] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb3670a2c69e4ffeadf61d88bf1ac51a in term 0.
I20260806 23:52:25.389263 22845 leader_election.cc:302] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aab8dac7f6de4764ac6c407661feccef, bb3670a2c69e4ffeadf61d88bf1ac51a; no voters:
I20260806 23:52:25.389389 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5e40e7ccba46b49f3f3a4f97fbca08" candidate_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" is_pre_election: true
I20260806 23:52:25.389569 23117 raft_consensus.cc:2454] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb3670a2c69e4ffeadf61d88bf1ac51a in term 0.
I20260806 23:52:25.389582 23418 raft_consensus.cc:2781] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:25.389631 23418 raft_consensus.cc:482] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:25.389671 23418 raft_consensus.cc:3037] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:25.390944 23418 raft_consensus.cc:504] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.391466 23418 leader_election.cc:288] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [CANDIDATE]: Term 1 election: Requested vote from peers 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841), aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121)
I20260806 23:52:25.391930 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5e40e7ccba46b49f3f3a4f97fbca08" candidate_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8"
I20260806 23:52:25.391963 23303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5e40e7ccba46b49f3f3a4f97fbca08" candidate_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab8dac7f6de4764ac6c407661feccef"
I20260806 23:52:25.392061 23117 raft_consensus.cc:3037] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:25.392098 23303 raft_consensus.cc:3037] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:25.393062 23117 raft_consensus.cc:2454] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb3670a2c69e4ffeadf61d88bf1ac51a in term 1.
I20260806 23:52:25.393390 22843 leader_election.cc:302] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec5831e2e0745a2b52a7e48ae1a9ae8, bb3670a2c69e4ffeadf61d88bf1ac51a; no voters:
I20260806 23:52:25.393431 23303 raft_consensus.cc:2454] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb3670a2c69e4ffeadf61d88bf1ac51a in term 1.
I20260806 23:52:25.393834 23418 raft_consensus.cc:2781] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:25.394163 23418 raft_consensus.cc:686] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 LEADER]: Becoming Leader. State: Replica: bb3670a2c69e4ffeadf61d88bf1ac51a, State: Running, Role: LEADER
I20260806 23:52:25.394454 23418 consensus_queue.cc:237] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } }
I20260806 23:52:25.397293 22716 catalog_manager.cc:5495] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129). New cstate: current_term: 1 leader_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb3670a2c69e4ffeadf61d88bf1ac51a" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 46319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 } health_report { overall_health: UNKNOWN } } }
I20260806 23:52:25.442798 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.438965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.10.225.129:46319)
0806 23:52:25.439326 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.439337 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.439353 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:52:25.439458 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.439461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.439521 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.439625 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:25.439936 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.440499 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.441311 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.441548 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.441638 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.441849 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.442337 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.442432 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.442546 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.442598 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":178,"threads_started":1}
I20260806 23:52:25.453860 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.450648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:52411 (local address 127.10.225.130:35841)
0806 23:52:25.450796 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.450801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.450911 (+ 110us) server_negotiation.cc:407] Connection header received
0806 23:52:25.451062 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.451064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.451111 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.451175 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:25.451442 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.451983 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.452820 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.453035 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.453122 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.453429 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.453502 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.453559 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.453651 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.453703 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:52:25.453915 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.450466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35841 (local address 127.10.225.129:52411)
0806 23:52:25.450818 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:25.450840 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:25.450954 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:25.451198 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:25.451201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:25.451209 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:52:25.451344 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.451350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.452084 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.452088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.452705 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.452713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.453161 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.453165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.453253 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.453297 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.453574 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.453582 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 23:52:25.453641 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 23:52:25.453719 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":121,"threads_started":1}
I20260806 23:52:25.459949 23117 raft_consensus.cc:1261] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Refusing update from remote peer bb3670a2c69e4ffeadf61d88bf1ac51a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:25.459990 23303 raft_consensus.cc:1261] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Refusing update from remote peer bb3670a2c69e4ffeadf61d88bf1ac51a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:25.460405 23514 consensus_queue.cc:1035] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec5831e2e0745a2b52a7e48ae1a9ae8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:25.460641 23418 consensus_queue.cc:1035] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab8dac7f6de4764ac6c407661feccef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:25.477530 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.473888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:46319 (local address 127.10.225.130:40079)
0806 23:52:25.474031 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:25.474050 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:25.474147 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:25.474381 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:25.474383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:25.474389 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 23:52:25.474516 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.474522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.475380 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.475384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.476187 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:25.476201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.476670 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.476674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:25.476776 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.476815 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.477165 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.477172 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 23:52:25.477249 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 23:52:25.477366 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 23:52:25.477942 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:25.474083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:40079 (local address 127.10.225.129:46319)
0806 23:52:25.474188 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:25.474193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:25.474206 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:52:25.474249 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:25.474252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:25.474299 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:25.474372 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:25.474630 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.475239 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.476326 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:25.476557 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:25.476643 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:25.476969 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:25.477059 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:25.477123 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:25.477744 (+ 621us) server_negotiation.cc:299] Negotiation successful
0806 23:52:25.477807 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:52:25.503276 22716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:25.503561 22716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:25.506549 22716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98507f57d3364114932cdf9a9d5a7920: Sending DeleteTablet for 3 replicas of tablet 5c5e40e7ccba46b49f3f3a4f97fbca08
I20260806 23:52:25.507615 22903 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5e40e7ccba46b49f3f3a4f97fbca08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:25 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:53458
I20260806 23:52:25.507776 23275 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5e40e7ccba46b49f3f3a4f97fbca08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:25 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:46596
I20260806 23:52:25.507866 23436 tablet_replica.cc:330] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: stopping tablet replica
I20260806 23:52:25.508005 23436 raft_consensus.cc:2229] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:52:25.508030 23435 tablet_replica.cc:330] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: stopping tablet replica
I20260806 23:52:25.508095 23091 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5e40e7ccba46b49f3f3a4f97fbca08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:25 UTC) from {username='oryx', principal='oryx/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:54088
I20260806 23:52:25.508279 23438 tablet_replica.cc:330] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: stopping tablet replica
I20260806 23:52:25.508388 23436 raft_consensus.cc:2258] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:25.508389 23438 raft_consensus.cc:2229] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:25.508391 23435 raft_consensus.cc:2229] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:25.508505 23438 raft_consensus.cc:2258] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:25.508656 23435 raft_consensus.cc:2258] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:25.509361 23436 ts_tablet_manager.cc:1905] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:25.509761 23435 ts_tablet_manager.cc:1905] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:25.511904 23436 ts_tablet_manager.cc:1918] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:52:25.511994 23436 log.cc:1196] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/5c5e40e7ccba46b49f3f3a4f97fbca08
I20260806 23:52:25.512255 23436 ts_tablet_manager.cc:1939] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P bb3670a2c69e4ffeadf61d88bf1ac51a: Deleting consensus metadata
I20260806 23:52:25.513123 22695 catalog_manager.cc:4841] TS bb3670a2c69e4ffeadf61d88bf1ac51a (127.10.225.129:46319): tablet 5c5e40e7ccba46b49f3f3a4f97fbca08 (table test-table [id=8690cd57a91b497e87f38903f956b17d]) successfully deleted
I20260806 23:52:25.513528 23435 ts_tablet_manager.cc:1918] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:52:25.513646 23435 log.cc:1196] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/5c5e40e7ccba46b49f3f3a4f97fbca08
I20260806 23:52:25.513943 23435 ts_tablet_manager.cc:1939] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P aab8dac7f6de4764ac6c407661feccef: Deleting consensus metadata
I20260806 23:52:25.514784 22698 catalog_manager.cc:4841] TS aab8dac7f6de4764ac6c407661feccef (127.10.225.131:46121): tablet 5c5e40e7ccba46b49f3f3a4f97fbca08 (table test-table [id=8690cd57a91b497e87f38903f956b17d]) successfully deleted
I20260806 23:52:25.516183 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 22819
I20260806 23:52:25.516813 23438 ts_tablet_manager.cc:1905] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:25.519579 23438 ts_tablet_manager.cc:1918] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 23:52:25.519660 23438 log.cc:1196] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/5c5e40e7ccba46b49f3f3a4f97fbca08
I20260806 23:52:25.519986 23438 ts_tablet_manager.cc:1939] T 5c5e40e7ccba46b49f3f3a4f97fbca08 P 7ec5831e2e0745a2b52a7e48ae1a9ae8: Deleting consensus metadata
I20260806 23:52:25.520805 22695 catalog_manager.cc:4841] TS 7ec5831e2e0745a2b52a7e48ae1a9ae8 (127.10.225.130:35841): tablet 5c5e40e7ccba46b49f3f3a4f97fbca08 (table test-table [id=8690cd57a91b497e87f38903f956b17d]) successfully deleted
I20260806 23:52:25.531800 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 23004
I20260806 23:52:25.549058 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 23190
I20260806 23:52:25.563552 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 22672
2026-08-06T23:52:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3459 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:52:25 dist-test-slave-nj21 krb5kdc[23532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:52:25 dist-test-slave-nj21 krb5kdc[23532](info): set up 2 sockets
Aug 06 23:52:25 dist-test-slave-nj21 krb5kdc[23532](info): commencing operation
krb5kdc: starting...
W20260806 23:52:26.741252 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:26 dist-test-slave-nj21 krb5kdc[23532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:52:26.881136 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:44503
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:46133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:27.014199 23549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:27.014525 23549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:27.014585 23549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:27.018929 23549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:27.019016 23549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:27.019045 23549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:27.019066 23549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:27.019086 23549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:27.024839 23549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:46133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:44503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:27.026081 23549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:27.027184 23549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:27.036749 23555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:27.036791 23549 server_base.cc:1034] running on GCE node
W20260806 23:52:27.036749 23557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:27.036751 23559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:27.037747 23549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:27.038980 23549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:27.042348 23549 hybrid_clock.cc:648] HybridClock initialized: now 1786060347042228 us; error 88 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T23:52:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1485 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:52:27 dist-test-slave-nj21 krb5kdc[23564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:52:27 dist-test-slave-nj21 krb5kdc[23564](info): set up 2 sockets
Aug 06 23:52:27 dist-test-slave-nj21 krb5kdc[23564](info): commencing operation
krb5kdc: starting...
W20260806 23:52:28.233708 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:28 dist-test-slave-nj21 krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:52:28.381289 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:40695
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:40695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:28.513023 23581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:28.513322 23581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:28.513366 23581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:28.517472 23581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:28.517549 23581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:28.517567 23581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:28.517583 23581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:28.517597 23581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:28.523188 23581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:40695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:40695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:28.524362 23581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:28.525400 23581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:28.534161 23592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:28.534178 23589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:28.534161 23590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:28.534276 23581 server_base.cc:1034] running on GCE node
I20260806 23:52:28.535006 23581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:28.536264 23581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:28.537451 23581 hybrid_clock.cc:648] HybridClock initialized: now 1786060348537435 us; error 30 us; skew 500 ppm
Aug 06 23:52:28 dist-test-slave-nj21 krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060348, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:28.545210 23581 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:52:28.546368 23581 webserver.cc:460] Webserver started at http://127.10.225.190:42557/ using document root <none> and password file <none>
I20260806 23:52:28.547050 23581 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:28.547118 23581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:28.547310 23581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:28.549736 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "41c424c2d05243f983c349ddf3b4e578"
format_stamp: "Formatted at 2026-08-06 23:52:28 on dist-test-slave-nj21"
server_key: "0e480a117429512fac31bcedd38e92b1"
server_key_iv: "d8c9dbaed57c6d06f67643a0ffcb4767"
server_key_version: "encryptionkey@0"
I20260806 23:52:28.550369 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "41c424c2d05243f983c349ddf3b4e578"
format_stamp: "Formatted at 2026-08-06 23:52:28 on dist-test-slave-nj21"
server_key: "0e480a117429512fac31bcedd38e92b1"
server_key_iv: "d8c9dbaed57c6d06f67643a0ffcb4767"
server_key_version: "encryptionkey@0"
I20260806 23:52:28.554787 23581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 23:52:28.557507 23601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:28.558930 23581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:28.559054 23581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "41c424c2d05243f983c349ddf3b4e578"
format_stamp: "Formatted at 2026-08-06 23:52:28 on dist-test-slave-nj21"
server_key: "0e480a117429512fac31bcedd38e92b1"
server_key_iv: "d8c9dbaed57c6d06f67643a0ffcb4767"
server_key_version: "encryptionkey@0"
I20260806 23:52:28.559173 23581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:28.585973 23581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:28.589483 23581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:28.589684 23581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:28.597618 23581 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:40695
I20260806 23:52:28.597635 23684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:40695 every 8 connection(s)
I20260806 23:52:28.599247 23581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:28.601372 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 23581
I20260806 23:52:28.601554 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:28.601895 11142 external_mini_cluster.cc:1428] Setting key 2462203b5e037b05861b96c7f9a4b89b
I20260806 23:52:28.602758 23685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:28.609992 23685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: Bootstrap starting.
Aug 06 23:52:28 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:28.612497 23685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:28.613350 23685 log.cc:824] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:28.615845 23685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: No bootstrap required, opened a new log
I20260806 23:52:28.617949 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:28.603528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.10.225.190:40695)
0806 23:52:28.604060 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:28.604083 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:28.604133 (+ 50us) server_negotiation.cc:407] Connection header received
0806 23:52:28.604968 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:28.604991 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:28.605456 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:28.605912 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:28.606896 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:28.608112 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:28.609139 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:28.609783 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:28.609965 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:28.612763 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:28.612772 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:28.612792 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:28.612831 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:28.614734 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:28.615290 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:28.615296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:28.615303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:28.615431 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:28.615746 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:28.615748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:28.615750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:28.616226 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:28.616382 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:28.616724 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 23:52:28.617054 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":174,"threads_started":1}
I20260806 23:52:28.619953 23685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } }
I20260806 23:52:28.620240 23685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:28.620307 23685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c424c2d05243f983c349ddf3b4e578, State: Initialized, Role: FOLLOWER
I20260806 23:52:28.620806 23685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } }
I20260806 23:52:28.620949 23685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:28.621016 23685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:28.621111 23685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:28.622141 23685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } }
I20260806 23:52:28.622478 23685 leader_election.cc:302] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41c424c2d05243f983c349ddf3b4e578; no voters:
I20260806 23:52:28.622978 23685 leader_election.cc:288] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:28.623132 23691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:28.623404 23691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 1 LEADER]: Becoming Leader. State: Replica: 41c424c2d05243f983c349ddf3b4e578, State: Running, Role: LEADER
I20260806 23:52:28.623765 23691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } }
I20260806 23:52:28.624333 23685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:28.625453 23694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c424c2d05243f983c349ddf3b4e578. Latest consensus state: current_term: 1 leader_uuid: "41c424c2d05243f983c349ddf3b4e578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } } }
I20260806 23:52:28.625607 23694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:28.625910 23693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41c424c2d05243f983c349ddf3b4e578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } } }
I20260806 23:52:28.626013 23693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:28.626157 23702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:28.630658 23702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:28.637017 23702 catalog_manager.cc:1347] Generated new cluster ID: 8c8d4108dadb4d26bb992e2fb9aa6ef3
I20260806 23:52:28.637111 23702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:28.648725 23702 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:28.649796 23702 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:28.655849 23702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: Generated new TSK 0
I20260806 23:52:28.657187 23702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:52:28.699016 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:40695
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:28.826079 23718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:28.826380 23718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:28.826445 23718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:28.830627 23718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:28.830708 23718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:28.830798 23718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:28.836390 23718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:40695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:28.837541 23718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:28.838603 23718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:28.849040 23726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:28.849061 23725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:28.849113 23718 server_base.cc:1034] running on GCE node
W20260806 23:52:28.849054 23728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:28.849913 23718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:28.850649 23718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:28.851814 23718 hybrid_clock.cc:648] HybridClock initialized: now 1786060348851798 us; error 37 us; skew 500 ppm
Aug 06 23:52:28 dist-test-slave-nj21 krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060348, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:28.854902 23718 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:52:28.855926 23718 webserver.cc:460] Webserver started at http://127.10.225.129:42441/ using document root <none> and password file <none>
I20260806 23:52:28.856506 23718 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:28.856585 23718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:28.856887 23718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:28.859141 23718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "b470365cd2aa4b5ba6c6541e1c410ddd"
format_stamp: "Formatted at 2026-08-06 23:52:28 on dist-test-slave-nj21"
server_key: "77e17afa64ef18f19d898e82d61a9dfd"
server_key_iv: "a7c2c2cdae53b6944c64a6e2da4135e9"
server_key_version: "encryptionkey@0"
I20260806 23:52:28.860234 23718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "b470365cd2aa4b5ba6c6541e1c410ddd"
format_stamp: "Formatted at 2026-08-06 23:52:28 on dist-test-slave-nj21"
server_key: "77e17afa64ef18f19d898e82d61a9dfd"
server_key_iv: "a7c2c2cdae53b6944c64a6e2da4135e9"
server_key_version: "encryptionkey@0"
I20260806 23:52:28.865314 23718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 23:52:28.868781 23736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:28.870309 23718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 23:52:28.870479 23718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "b470365cd2aa4b5ba6c6541e1c410ddd"
format_stamp: "Formatted at 2026-08-06 23:52:28 on dist-test-slave-nj21"
server_key: "77e17afa64ef18f19d898e82d61a9dfd"
server_key_iv: "a7c2c2cdae53b6944c64a6e2da4135e9"
server_key_version: "encryptionkey@0"
I20260806 23:52:28.870617 23718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:28.883253 23718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:28.886548 23718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:28.886744 23718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:28.887477 23718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:28.888482 23718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:28.888535 23718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:28.888590 23718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:28.888619 23718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:28.897722 23718 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:40909
I20260806 23:52:28.897751 23892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:40909 every 8 connection(s)
I20260806 23:52:28.898752 23718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:28.899323 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 23718
I20260806 23:52:28.899601 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:28.900084 11142 external_mini_cluster.cc:1428] Setting key 5dcb50d04ec532dbb7a3a4a8fc30b7d7
Aug 06 23:52:28 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060348, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:52:28.914546 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:28.900796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:37049 (local address 127.10.225.190:40695)
0806 23:52:28.900953 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:28.900958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:28.901938 (+ 980us) server_negotiation.cc:407] Connection header received
0806 23:52:28.903491 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:28.903495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:28.903557 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:28.903634 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:28.904889 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:28.905657 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:28.906940 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:28.907188 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:28.907282 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:28.910603 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:28.910607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:28.910610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:28.910650 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:28.912264 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:28.912868 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:28.912872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:28.912874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:28.912961 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:28.913306 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:28.913309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:28.913311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:28.913526 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:28.913621 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:28.914205 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 23:52:28.914366 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 23:52:28.915409 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:28.901248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:37049)
0806 23:52:28.901771 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:28.901825 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:28.903193 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:28.903844 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:28.903852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:28.904235 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:28.904739 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:28.904753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:28.905810 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:28.905823 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:28.906806 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:28.906825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:28.907351 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:28.907356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:28.907525 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:28.908137 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:28.908154 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:28.910262 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:28.912436 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:28.912445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:28.912464 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:28.912757 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:28.913050 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:28.913054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:28.913056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:28.913214 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:28.913671 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:28.913678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:28.913959 (+ 281us) client_negotiation.cc:769] Sending connection context
0806 23:52:28.914141 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 23:52:28.914507 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260806 23:52:28.916811 23894 heartbeater.cc:347] Connected to a master server at 127.10.225.190:40695
I20260806 23:52:28.917052 23894 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:28.917575 23894 heartbeater.cc:511] Master 127.10.225.190:40695 requested a full tablet report, sending...
I20260806 23:52:28.919149 23630 ts_manager.cc:194] Registered new tserver with Master: b470365cd2aa4b5ba6c6541e1c410ddd (127.10.225.129:40909)
I20260806 23:52:28.920336 23630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:37049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:52:28.931586 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:40695
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:29.057821 23900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:29.058120 23900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:29.058166 23900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:29.062476 23900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:29.062565 23900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:29.062630 23900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:29.069768 23900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:40695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:29.070923 23900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:29.072041 23900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:29.082160 23907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:29.082160 23908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:29.082183 23911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:29.082446 23900 server_base.cc:1034] running on GCE node
I20260806 23:52:29.082996 23900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:29.083727 23900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:29.084929 23900 hybrid_clock.cc:648] HybridClock initialized: now 1786060349084896 us; error 66 us; skew 500 ppm
Aug 06 23:52:29 dist-test-slave-nj21 krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060349, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:29.088617 23900 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:52:29.089793 23900 webserver.cc:460] Webserver started at http://127.10.225.130:46869/ using document root <none> and password file <none>
I20260806 23:52:29.090409 23900 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:29.090481 23900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:29.090705 23900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:29.093055 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "25cf7740a3ab49ad8f169e6c533e36ee"
format_stamp: "Formatted at 2026-08-06 23:52:29 on dist-test-slave-nj21"
server_key: "d2b590e20b41d1c764b8cf0dc2f232dd"
server_key_iv: "770187236ec54170d098e23f4d8ef2f6"
server_key_version: "encryptionkey@0"
I20260806 23:52:29.093640 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "25cf7740a3ab49ad8f169e6c533e36ee"
format_stamp: "Formatted at 2026-08-06 23:52:29 on dist-test-slave-nj21"
server_key: "d2b590e20b41d1c764b8cf0dc2f232dd"
server_key_iv: "770187236ec54170d098e23f4d8ef2f6"
server_key_version: "encryptionkey@0"
I20260806 23:52:29.097752 23900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:52:29.101053 23919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.103013 23900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:52:29.103199 23900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "25cf7740a3ab49ad8f169e6c533e36ee"
format_stamp: "Formatted at 2026-08-06 23:52:29 on dist-test-slave-nj21"
server_key: "d2b590e20b41d1c764b8cf0dc2f232dd"
server_key_iv: "770187236ec54170d098e23f4d8ef2f6"
server_key_version: "encryptionkey@0"
I20260806 23:52:29.103351 23900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:29.122076 23900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:29.126639 23900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:29.126905 23900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:29.127720 23900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:29.128835 23900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:29.128891 23900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.128957 23900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:29.128981 23900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.140519 23900 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:40465
I20260806 23:52:29.140547 24079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:40465 every 8 connection(s)
I20260806 23:52:29.141688 23900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:29.150590 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 23900
I20260806 23:52:29.150712 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:29.151000 11142 external_mini_cluster.cc:1428] Setting key f89fbac8216bfbed4e92e527e8d818f7
Aug 06 23:52:29 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060349, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:29.158227 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:29.144157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:46161 (local address 127.10.225.190:40695)
0806 23:52:29.144412 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:29.144418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:29.145886 (+ 1468us) server_negotiation.cc:407] Connection header received
0806 23:52:29.146988 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:29.146992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:29.147075 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:29.147157 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:29.149101 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:29.149679 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:29.151045 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:29.151300 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:29.151397 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:29.154253 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:29.154257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:29.154259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:29.154285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:29.155944 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:29.156514 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:29.156517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:29.156519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:29.156614 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:29.156958 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:29.156961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:29.156962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:29.157204 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:29.157300 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:29.157885 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 23:52:29.158039 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260806 23:52:29.159188 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:29.144716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:46161)
0806 23:52:29.145672 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:29.145726 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:29.146737 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:29.147386 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:29.147395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:29.147964 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:29.148802 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:29.148840 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:29.149824 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:29.149828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:29.150904 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:29.150918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:29.151466 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:29.151471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:29.151645 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:29.152428 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:29.152452 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:29.154073 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:29.156089 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:29.156097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:29.156118 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:29.156417 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:29.156704 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:29.156707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:29.156708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:29.156869 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:29.157307 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:29.157313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:29.157615 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 23:52:29.157794 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 23:52:29.158171 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":179,"threads_started":1}
I20260806 23:52:29.160466 24082 heartbeater.cc:347] Connected to a master server at 127.10.225.190:40695
I20260806 23:52:29.160727 24082 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:29.161274 24082 heartbeater.cc:511] Master 127.10.225.190:40695 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:52:29.162811 23630 ts_manager.cc:194] Registered new tserver with Master: 25cf7740a3ab49ad8f169e6c533e36ee (127.10.225.130:40465)
I20260806 23:52:29.163434 23630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:46161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:52:29.188091 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:40695
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:29.323177 24087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:29.323585 24087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:29.323808 24087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:29.330245 24087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:29.330346 24087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:29.330418 24087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:29.337656 24087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:40695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:29.338842 24087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:29.339955 24087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:52:29.351092 24087 server_base.cc:1034] running on GCE node
W20260806 23:52:29.351027 24096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:29.351027 24095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:29.351027 24098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:29.352020 24087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:29.352747 24087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:29.353937 24087 hybrid_clock.cc:648] HybridClock initialized: now 1786060349353915 us; error 31 us; skew 500 ppm
Aug 06 23:52:29 dist-test-slave-nj21 krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060349, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:29.357003 24087 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:52:29.358032 24087 webserver.cc:460] Webserver started at http://127.10.225.131:46715/ using document root <none> and password file <none>
I20260806 23:52:29.359225 24087 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:29.359357 24087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:29.359611 24087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:29.361822 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "b264c1e76e71415fa86553da752802e6"
format_stamp: "Formatted at 2026-08-06 23:52:29 on dist-test-slave-nj21"
server_key: "9ae710cebdd73e39d48da123f4314029"
server_key_iv: "c670afeedcb0482c0f971562e8e11199"
server_key_version: "encryptionkey@0"
I20260806 23:52:29.362460 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "b264c1e76e71415fa86553da752802e6"
format_stamp: "Formatted at 2026-08-06 23:52:29 on dist-test-slave-nj21"
server_key: "9ae710cebdd73e39d48da123f4314029"
server_key_iv: "c670afeedcb0482c0f971562e8e11199"
server_key_version: "encryptionkey@0"
I20260806 23:52:29.366864 24087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 23:52:29.370417 24105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.372056 24087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:29.372188 24087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "b264c1e76e71415fa86553da752802e6"
format_stamp: "Formatted at 2026-08-06 23:52:29 on dist-test-slave-nj21"
server_key: "9ae710cebdd73e39d48da123f4314029"
server_key_iv: "c670afeedcb0482c0f971562e8e11199"
server_key_version: "encryptionkey@0"
I20260806 23:52:29.372309 24087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:29.384680 24087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:29.387920 24087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:29.388115 24087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:29.388758 24087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:29.389683 24087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:29.389732 24087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.389787 24087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:29.389817 24087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.399847 24087 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:42451
I20260806 23:52:29.399881 24261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:42451 every 8 connection(s)
I20260806 23:52:29.400818 24087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:29.407370 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 24087
I20260806 23:52:29.407507 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:29.407832 11142 external_mini_cluster.cc:1428] Setting key b0cd3ae497fd1413fea78b09de1b6a03
Aug 06 23:52:29 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060349, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:29.416456 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:29.403441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:46455 (local address 127.10.225.190:40695)
0806 23:52:29.403616 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:29.403620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:29.404536 (+ 916us) server_negotiation.cc:407] Connection header received
0806 23:52:29.405840 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:29.405844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:29.405911 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:29.405992 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:29.407403 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:29.408347 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:29.409868 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:29.410102 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:29.410190 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:29.412799 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:29.412802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:29.412805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:29.412831 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:29.414237 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:29.414777 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:29.414780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:29.414782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:29.414859 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:29.415201 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:29.415205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:29.415206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:29.415409 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:29.415508 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:29.416104 (+ 596us) server_negotiation.cc:299] Negotiation successful
0806 23:52:29.416240 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 23:52:29.417326 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:29.403759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:46455)
0806 23:52:29.404346 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:29.404392 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:29.405601 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:29.406199 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:29.406205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:29.406587 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:29.407158 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:29.407179 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:29.408499 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:29.408503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:29.409745 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:29.409763 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:29.410243 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:29.410248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:29.410438 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:29.411062 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:29.411080 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:29.412624 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:29.414372 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:29.414380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:29.414400 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:29.414675 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:29.414943 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:29.414946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:29.414956 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:29.415108 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:29.415514 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:29.415521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:29.415850 (+ 329us) client_negotiation.cc:769] Sending connection context
0806 23:52:29.416046 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 23:52:29.416424 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":137,"threads_started":1}
I20260806 23:52:29.418701 24262 heartbeater.cc:347] Connected to a master server at 127.10.225.190:40695
I20260806 23:52:29.418951 24262 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:29.419452 24262 heartbeater.cc:511] Master 127.10.225.190:40695 requested a full tablet report, sending...
I20260806 23:52:29.420629 23630 ts_manager.cc:194] Registered new tserver with Master: b264c1e76e71415fa86553da752802e6 (127.10.225.131:42451)
I20260806 23:52:29.421298 23630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:46455
I20260806 23:52:29.422112 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab.
Entry for principal oryx/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab.
WARNING: no policy specified for oryx/127.10.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab.
Entry for principal oryx/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab.
WARNING: no policy specified for oryx/127.10.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab.
Entry for principal oryx/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab.
I20260806 23:52:29.512773 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 23718
I20260806 23:52:29.524861 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 23900
I20260806 23:52:29.552337 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 24087
I20260806 23:52:29.565286 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 23581
I20260806 23:52:29.578564 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:40695
--webserver_interface=127.10.225.190
--webserver_port=42557
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:40695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:29.709432 24272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:29.709755 24272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:29.709803 24272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:29.714725 24272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:29.714847 24272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:29.714869 24272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:29.714888 24272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:29.714903 24272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:29.722424 24272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:40695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:40695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=42557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:29.723752 24272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:29.724870 24272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:29.733675 24280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:29.733740 24283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:29.733675 24279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:29.734170 24272 server_base.cc:1034] running on GCE node
I20260806 23:52:29.734776 24272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:29.735997 24272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:29.737195 24272 hybrid_clock.cc:648] HybridClock initialized: now 1786060349737177 us; error 31 us; skew 500 ppm
Aug 06 23:52:29 dist-test-slave-nj21 krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060349, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:29.740248 24272 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:52:29.741189 24272 webserver.cc:460] Webserver started at http://127.10.225.190:42557/ using document root <none> and password file <none>
I20260806 23:52:29.741776 24272 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:29.741854 24272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:29.746343 24272 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:52:29.749454 24291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.751080 24272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:52:29.751274 24272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "41c424c2d05243f983c349ddf3b4e578"
format_stamp: "Formatted at 2026-08-06 23:52:28 on dist-test-slave-nj21"
server_key: "0e480a117429512fac31bcedd38e92b1"
server_key_iv: "d8c9dbaed57c6d06f67643a0ffcb4767"
server_key_version: "encryptionkey@0"
I20260806 23:52:29.751962 24272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:29.776715 24272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:29.780797 24272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:29.781009 24272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:29.788858 24272 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:40695
I20260806 23:52:29.788901 24353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:40695 every 8 connection(s)
I20260806 23:52:29.790073 24272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:29.794740 24354 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:52:29.796738 24354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: Bootstrap starting.
I20260806 23:52:29.797402 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 24272
I20260806 23:52:29.797859 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:40909
--local_ip_for_outbound_sockets=127.10.225.129
--tserver_master_addrs=127.10.225.190:40695
--webserver_port=42441
--webserver_interface=127.10.225.129
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.225.129
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:52:29.816896 24354 log.cc:824] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:29.826443 24354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:52:29.826907 24354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: Bootstrap complete.
I20260806 23:52:29.832173 24354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } }
I20260806 23:52:29.832793 24354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c424c2d05243f983c349ddf3b4e578, State: Initialized, Role: FOLLOWER
I20260806 23:52:29.833635 24354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } }
I20260806 23:52:29.833925 24354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:29.834017 24354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:29.834147 24354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:52:29.836549 24354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } }
I20260806 23:52:29.836949 24354 leader_election.cc:302] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41c424c2d05243f983c349ddf3b4e578; no voters:
I20260806 23:52:29.837321 24354 leader_election.cc:288] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 23:52:29.837448 24360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 23:52:29.837693 24360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [term 2 LEADER]: Becoming Leader. State: Replica: 41c424c2d05243f983c349ddf3b4e578, State: Running, Role: LEADER
I20260806 23:52:29.838101 24360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } }
I20260806 23:52:29.838764 24354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:29.839795 24362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c424c2d05243f983c349ddf3b4e578. Latest consensus state: current_term: 2 leader_uuid: "41c424c2d05243f983c349ddf3b4e578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } } }
I20260806 23:52:29.839941 24362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:29.840313 24361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "41c424c2d05243f983c349ddf3b4e578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c424c2d05243f983c349ddf3b4e578" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 40695 } } }
I20260806 23:52:29.840404 24361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:29.845274 24384 catalog_manager.cc:1259] Loaded cluster ID: 8c8d4108dadb4d26bb992e2fb9aa6ef3
I20260806 23:52:29.845364 24384 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: loading cluster ID for follower catalog manager: success
I20260806 23:52:29.847851 24384 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: acquiring CA information for follower catalog manager: success
I20260806 23:52:29.849341 24384 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:52:29.849504 24373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:29.851199 24373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:29.851716 24373 catalog_manager.cc:1259] Loaded cluster ID: 8c8d4108dadb4d26bb992e2fb9aa6ef3
I20260806 23:52:29.851774 24373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:29.852448 24373 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:29.852897 24373 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 41c424c2d05243f983c349ddf3b4e578: Loaded TSK: 0
I20260806 23:52:29.853729 24373 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 23:52:29.920423 24356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:29.920697 24356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:29.920744 24356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:29.924860 24356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:29.924929 24356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:29.925001 24356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:29.930502 24356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.129.keytab
--principal=oryx/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:40909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=42441
--tserver_master_addrs=127.10.225.190:40695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:29.931540 24356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:29.932564 24356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:29.943286 24392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:29.943306 24391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:29.943295 24356 server_base.cc:1034] running on GCE node
W20260806 23:52:29.943347 24394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:29.944334 24356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:29.945262 24356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:29.946497 24356 hybrid_clock.cc:648] HybridClock initialized: now 1786060349946445 us; error 35 us; skew 500 ppm
Aug 06 23:52:29 dist-test-slave-nj21 krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060349, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:29.950165 24356 init.cc:377] Logged in from keytab as oryx/127.10.225.129@KRBTEST.COM (short username oryx)
I20260806 23:52:29.951261 24356 webserver.cc:460] Webserver started at http://127.10.225.129:42441/ using document root <none> and password file <none>
I20260806 23:52:29.951900 24356 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:29.951993 24356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:29.956401 24356 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:29.958766 24403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.960807 24356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:29.961033 24356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "b470365cd2aa4b5ba6c6541e1c410ddd"
format_stamp: "Formatted at 2026-08-06 23:52:28 on dist-test-slave-nj21"
server_key: "77e17afa64ef18f19d898e82d61a9dfd"
server_key_iv: "a7c2c2cdae53b6944c64a6e2da4135e9"
server_key_version: "encryptionkey@0"
I20260806 23:52:29.961813 24356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:29.979763 24356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:29.982992 24356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:29.983184 24356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:29.983839 24356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:29.985253 24356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:29.985348 24356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.985428 24356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:29.985455 24356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:29.995913 24356 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:40909
I20260806 23:52:29.996021 24554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:40909 every 8 connection(s)
I20260806 23:52:29.997004 24356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:30.006582 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 24356
I20260806 23:52:30.007244 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:40465
--local_ip_for_outbound_sockets=127.10.225.130
--tserver_master_addrs=127.10.225.190:40695
--webserver_port=46869
--webserver_interface=127.10.225.130
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.225.130
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 23:52:30.010922 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:29.999505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:34841 (local address 127.10.225.190:40695)
0806 23:52:30.000051 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.000071 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.000436 (+ 365us) server_negotiation.cc:407] Connection header received
0806 23:52:30.001928 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.001938 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.002224 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.002518 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.003715 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.004682 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.005803 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.006288 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.006457 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.010153 (+ 3696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:34841: BlockingRecv error: recv got EOF from 127.10.225.129:34841 (error 108)
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":165,"threads_started":1}
I20260806 23:52:30.010936 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:29.999734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:34841)
0806 23:52:30.000250 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:30.000294 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:30.001530 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:30.002622 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:30.002631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:30.003070 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:30.003525 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.003540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.004817 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.004820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.005678 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.005689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.006466 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.006470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.006640 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.007451 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:30.007474 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:30.009917 (+ 2443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":129,"threads_started":1}
W20260806 23:52:30.011425 24557 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:40695 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.225.190:40695: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:30.018144 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:30.011907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:54197)
0806 23:52:30.012009 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:30.012022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:30.012093 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:30.012399 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:30.012402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:30.012554 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:30.012651 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.012657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.013701 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.013705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.014385 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.014399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.014935 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.014939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.015039 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.015761 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:30.015776 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:30.017957 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 23:52:30.018286 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:30.012089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:54197 (local address 127.10.225.190:40695)
0806 23:52:30.012210 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.012213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.012224 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:52:30.012257 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.012259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.012299 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.012366 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.012758 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.013460 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.014532 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.014800 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.014902 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.018167 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:54197: BlockingRecv error: recv got EOF from 127.10.225.129:54197 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:30.024145 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:30.018908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:41927)
0806 23:52:30.019022 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:30.019038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:30.019152 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:30.019400 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:30.019403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:30.019573 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:30.019660 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.019666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.020425 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.020428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.020962 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.020970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.021391 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.021394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.021469 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.022050 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:30.022066 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:30.023974 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 23:52:30.024327 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:30.019029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:41927 (local address 127.10.225.190:40695)
0806 23:52:30.019187 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.019192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.019218 (+ 26us) server_negotiation.cc:407] Connection header received
0806 23:52:30.019259 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.019262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.019306 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.019368 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.019780 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.020320 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.021074 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.021289 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.021377 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.024190 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:41927: BlockingRecv error: recv got EOF from 127.10.225.129:41927 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 23:52:30.024534 24557 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:52:30.133167 24562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:30.133481 24562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:30.133545 24562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:30.137703 24562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:30.137785 24562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:30.137880 24562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:30.145589 24562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.130.keytab
--principal=oryx/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:40465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=46869
--tserver_master_addrs=127.10.225.190:40695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:30.147281 24562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:30.148875 24562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:30.161358 24567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:30.161367 24568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:30.161489 24562 server_base.cc:1034] running on GCE node
W20260806 23:52:30.161358 24571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:30.162160 24562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:30.162846 24562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:30.164112 24562 hybrid_clock.cc:648] HybridClock initialized: now 1786060350164061 us; error 40 us; skew 500 ppm
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060350, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:30.167986 24562 init.cc:377] Logged in from keytab as oryx/127.10.225.130@KRBTEST.COM (short username oryx)
I20260806 23:52:30.169034 24562 webserver.cc:460] Webserver started at http://127.10.225.130:46869/ using document root <none> and password file <none>
I20260806 23:52:30.169643 24562 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:30.169732 24562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:30.173983 24562 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:52:30.177578 24580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:30.179102 24562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:52:30.179251 24562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "25cf7740a3ab49ad8f169e6c533e36ee"
format_stamp: "Formatted at 2026-08-06 23:52:29 on dist-test-slave-nj21"
server_key: "d2b590e20b41d1c764b8cf0dc2f232dd"
server_key_iv: "770187236ec54170d098e23f4d8ef2f6"
server_key_version: "encryptionkey@0"
I20260806 23:52:30.179934 24562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:30.194708 24562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:30.198060 24562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:30.198254 24562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:30.198889 24562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:30.199842 24562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:30.199892 24562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:30.199949 24562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:30.199977 24562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:30.209228 24562 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:40465
I20260806 23:52:30.209262 24740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:40465 every 8 connection(s)
I20260806 23:52:30.210357 24562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:30.217314 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 24562
I20260806 23:52:30.217778 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:42451
--local_ip_for_outbound_sockets=127.10.225.131
--tserver_master_addrs=127.10.225.190:40695
--webserver_port=46715
--webserver_interface=127.10.225.131
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.225.131
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 23:52:30.222175 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:30.212375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:34831 (local address 127.10.225.190:40695)
0806 23:52:30.212543 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.212547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.213584 (+ 1037us) server_negotiation.cc:407] Connection header received
0806 23:52:30.214470 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.214473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.214541 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.214614 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.215848 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.216391 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.217536 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.217757 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.217845 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.222034 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:34831: BlockingRecv error: recv got EOF from 127.10.225.130:34831 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 23:52:30.223115 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:30.212739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:34831)
0806 23:52:30.213421 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:30.213462 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:30.214263 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:30.214797 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:30.214804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:30.215175 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:30.215667 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.215719 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.216515 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.216518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.217409 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.217422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.217911 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.217916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.218089 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.218663 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:30.218680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:30.221855 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":123,"threads_started":1}
W20260806 23:52:30.223986 24741 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:40695 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.225.190:40695: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:30.231063 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:30.224830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:51487)
0806 23:52:30.224974 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:30.224993 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:30.225088 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:30.225371 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:30.225374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:30.225556 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:30.225676 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.225681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.226693 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.226698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.227351 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.227365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.227919 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.227922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.228024 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.228684 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:30.228698 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:30.230907 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 23:52:30.231211 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:30.224997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:51487 (local address 127.10.225.190:40695)
0806 23:52:30.225149 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.225154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.225168 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:30.225226 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.225228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.225268 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.225335 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.225804 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.226478 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.227491 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.227786 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.227886 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.231096 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:51487: BlockingRecv error: recv got EOF from 127.10.225.130:51487 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:30.237444 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:30.231959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:46657)
0806 23:52:30.232064 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:30.232078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:30.232157 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:30.232426 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:30.232428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:30.232555 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:30.232644 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.232649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.233725 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.233728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.234275 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.234340 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.234929 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.234933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.235006 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.235650 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:30.235663 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:30.237328 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 23:52:30.237715 24741 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:52:30.237766 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:30.232050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:46657 (local address 127.10.225.190:40695)
0806 23:52:30.232167 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.232171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.232216 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:52:30.232262 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.232265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.232324 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.232396 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.232733 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.233587 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.234450 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.234797 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.234893 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.237507 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:46657: BlockingRecv error: recv got EOF from 127.10.225.130:46657 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 23:52:30.345491 24745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:30.345768 24745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:30.345813 24745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:30.349982 24745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:30.350063 24745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:30.350128 24745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:30.355705 24745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/krb5kdc/oryx_127.10.225.131.keytab
--principal=oryx/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:42451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=46715
--tserver_master_addrs=127.10.225.190:40695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:30.356969 24745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:30.358107 24745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:30.368903 24753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:30.368916 24752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:30.368908 24755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:30.369325 24745 server_base.cc:1034] running on GCE node
I20260806 23:52:30.369830 24745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:30.370533 24745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:30.371812 24745 hybrid_clock.cc:648] HybridClock initialized: now 1786060350371767 us; error 43 us; skew 500 ppm
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060350, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:30.375510 24745 init.cc:377] Logged in from keytab as oryx/127.10.225.131@KRBTEST.COM (short username oryx)
I20260806 23:52:30.376750 24745 webserver.cc:460] Webserver started at http://127.10.225.131:46715/ using document root <none> and password file <none>
I20260806 23:52:30.377616 24745 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:30.377882 24745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:30.382710 24745 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 23:52:30.385519 24768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:30.387444 24745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:30.387758 24745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "b264c1e76e71415fa86553da752802e6"
format_stamp: "Formatted at 2026-08-06 23:52:29 on dist-test-slave-nj21"
server_key: "9ae710cebdd73e39d48da123f4314029"
server_key_iv: "c670afeedcb0482c0f971562e8e11199"
server_key_version: "encryptionkey@0"
I20260806 23:52:30.388424 24745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:30.411963 24745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:30.415894 24745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:30.416124 24745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:30.416913 24745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:30.418005 24745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:30.418062 24745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:30.418126 24745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:30.418154 24745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:30.429246 24745 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:42451
I20260806 23:52:30.429270 24930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:42451 every 8 connection(s)
I20260806 23:52:30.431123 24745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:30.438141 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 24745
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 23:52:30.445091 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:30.433406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:60325 (local address 127.10.225.190:40695)
0806 23:52:30.433565 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.433569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.434666 (+ 1097us) server_negotiation.cc:407] Connection header received
0806 23:52:30.436034 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.436037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.436097 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.436173 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.437481 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.438101 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.439925 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.440376 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.440464 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.444926 (+ 4462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:60325: BlockingRecv error: recv got EOF from 127.10.225.131:60325 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:52:30.445978 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:30.433840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:60325)
0806 23:52:30.434500 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:30.434548 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:30.435804 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:30.436387 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:30.436394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:30.436794 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:30.437322 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.437336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.438363 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.438367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.439800 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.439811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.440562 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.440567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.440800 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.441561 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:30.441589 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:30.444752 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":181,"threads_started":1}
W20260806 23:52:30.446754 24931 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:40695 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.225.190:40695: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
I20260806 23:52:30.449487 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:30.439527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.10.225.190:40695)
0806 23:52:30.439850 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.439853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.439867 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:30.439919 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.439922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.440070 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.440198 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.441184 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.441798 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.442892 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.443163 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.443254 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.444640 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:30.444646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:30.444667 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:30.444696 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:30.446587 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:30.447098 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:30.447105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:30.447111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:30.447278 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:30.447867 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:30.447870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:30.447871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:30.448397 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:30.448628 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:30.449015 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 23:52:30.449320 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":99,"threads_started":1}
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:30.454085 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:30.447469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:60267)
0806 23:52:30.447638 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:30.447654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:30.447775 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:30.448109 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:30.448111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:30.448254 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:30.448357 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.448361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.449095 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.449098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.450086 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.450106 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.450891 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.450894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.450985 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.451543 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:30.451557 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:30.453962 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 23:52:30.454383 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:30.447524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:60267 (local address 127.10.225.190:40695)
0806 23:52:30.447737 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.447742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.447758 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:52:30.447891 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.447894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.447939 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.447979 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.448475 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.448992 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.450242 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.450524 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.450748 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.454158 (+ 3410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:60267: BlockingRecv error: recv got EOF from 127.10.225.131:60267 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:30 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:30.460577 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:30.454708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:38285)
0806 23:52:30.454843 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:30.454859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:30.454942 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:30.455216 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:30.455220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:30.455400 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:30.455522 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.455631 (+ 109us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.456591 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.456595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.457216 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:30.457226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.457759 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.457761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:30.457827 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.458316 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:30.458329 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:30.460437 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 23:52:30.460733 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:30.454784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:38285 (local address 127.10.225.190:40695)
0806 23:52:30.454917 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:30.454920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:30.454933 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:52:30.455009 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:30.455012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:30.455049 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:30.455111 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:30.455819 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.456431 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.457350 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:30.457623 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:30.457725 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:30.460618 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:38285: BlockingRecv error: recv got EOF from 127.10.225.131:38285 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 23:52:30.460907 24931 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 23:52:31 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:31 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:31.031745 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:31.025277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:57711)
0806 23:52:31.025695 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:31.025716 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:31.025889 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:31.026291 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:31.026294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:31.026500 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:31.026644 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:31.026651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.027622 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.027625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:31.028270 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:31.028280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.028941 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.028944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:31.029029 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:31.029606 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:31.029619 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:31.031586 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":118,"threads_started":1}
W20260806 23:52:31.031999 24937 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:31.025444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:57711 (local address 127.10.225.190:40695)
0806 23:52:31.025745 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:31.025750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:31.025835 (+ 85us) server_negotiation.cc:407] Connection header received
0806 23:52:31.026011 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:31.026013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:31.026085 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:31.026200 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:31.026810 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.027483 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.028486 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.028796 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.028908 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:31.031851 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:57711: BlockingRecv error: recv got EOF from 127.10.225.129:57711 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
Aug 06 23:52:31 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:31 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:31.244004 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:31.238345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:59371)
0806 23:52:31.238684 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:31.238706 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:31.238813 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:31.239056 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:31.239059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:31.239285 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:31.239420 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:31.239427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.240280 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.240284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:31.240937 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:31.240947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.241375 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.241379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:31.241470 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:31.241998 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:31.242013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:31.243876 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
W20260806 23:52:31.244148 24937 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:31.238596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:59371 (local address 127.10.225.190:40695)
0806 23:52:31.238762 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:31.238766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:31.238781 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:52:31.238902 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:31.238905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:31.238958 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:31.239026 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:31.239590 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.240173 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.241045 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.241272 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.241358 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:31.244019 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:59371: BlockingRecv error: recv got EOF from 127.10.225.130:59371 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 23:52:31 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:31 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:31.468621 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:31.461672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:37505)
0806 23:52:31.461978 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:31.462138 (+ 160us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:31.462336 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:31.462771 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:31.462774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:31.463005 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:31.463121 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:31.463127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.464036 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.464039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:31.464709 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:31.464721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.465205 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.465208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:31.465273 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:31.465892 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:31.465915 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:31.468408 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
W20260806 23:52:31.468739 24937 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:31.461800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:37505 (local address 127.10.225.190:40695)
0806 23:52:31.461982 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:31.461986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:31.462365 (+ 379us) server_negotiation.cc:407] Connection header received
0806 23:52:31.462428 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:31.462430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:31.462595 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:31.462681 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:31.463327 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.463904 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.464844 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:31.465078 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:31.465165 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:31.468597 (+ 3432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:37505: BlockingRecv error: recv got EOF from 127.10.225.131:37505 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 23:52:32 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:32 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:32.039000 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:32.032796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:46557)
0806 23:52:32.033122 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:32.033136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:32.033230 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:32.033477 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:32.033479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:32.033663 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:32.033757 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:32.033762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.034597 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.034600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:32.035216 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:32.035228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.035781 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.035785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:32.035868 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:32.036437 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:32.036449 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:32.038828 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
W20260806 23:52:32.039095 24941 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:32.032934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:46557 (local address 127.10.225.190:40695)
0806 23:52:32.033168 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:32.033172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:32.033188 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:52:32.033313 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:32.033316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:32.033360 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:32.033455 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:32.033919 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.034450 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.035361 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.035643 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.035753 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:32.038960 (+ 3207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:46557: BlockingRecv error: recv got EOF from 127.10.225.129:46557 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
Aug 06 23:52:32 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:32 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:32.252018 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:32.245085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:52235)
0806 23:52:32.245590 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:32.245609 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:32.245784 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:32.246076 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:32.246079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:32.246349 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:32.246475 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:32.246481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.247341 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.247345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:32.248074 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:32.248085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.248699 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.248702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:32.248791 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:32.249452 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:32.249484 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:32.251863 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":128,"threads_started":1}
W20260806 23:52:32.252180 24941 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:32.245286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:52235 (local address 127.10.225.190:40695)
0806 23:52:32.245521 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:32.245527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:32.245711 (+ 184us) server_negotiation.cc:407] Connection header received
0806 23:52:32.245893 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:32.245896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:32.245953 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:32.246030 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:32.246598 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.247229 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.248237 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.248541 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.248665 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:32.252056 (+ 3391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:52235: BlockingRecv error: recv got EOF from 127.10.225.130:52235 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 23:52:32 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:32 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:32.476028 24943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:32.469742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:41229)
0806 23:52:32.470086 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:32.470101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:32.470242 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:32.470621 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:32.470624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:32.470840 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:32.470959 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:32.470965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.471794 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.471798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:32.472371 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:32.472381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.472857 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.472860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:32.472936 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:32.473564 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:32.473577 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:32.475842 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
W20260806 23:52:32.476419 24941 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:32.469890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:41229 (local address 127.10.225.190:40695)
0806 23:52:32.470031 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:32.470035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:32.470188 (+ 153us) server_negotiation.cc:407] Connection header received
0806 23:52:32.470357 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:32.470360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:32.470423 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:32.470509 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:32.471078 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.471624 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.472505 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:32.472732 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:32.472825 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:32.476265 (+ 3440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:41229: BlockingRecv error: recv got EOF from 127.10.225.131:41229 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 23:52:33 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:33 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:33.047032 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:33.040245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:43889)
0806 23:52:33.040599 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:33.040613 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:33.040734 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:33.041123 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:33.041127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:33.041335 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:33.041460 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:33.041466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.042387 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.042390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:33.043000 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:33.043009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.043564 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.043568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:33.043658 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:33.044311 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:33.044325 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:33.046867 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
W20260806 23:52:33.047329 24947 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:33.040269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:43889 (local address 127.10.225.190:40695)
0806 23:52:33.040605 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:33.040609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:33.040676 (+ 67us) server_negotiation.cc:407] Connection header received
0806 23:52:33.040819 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:33.040821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:33.040869 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:33.040993 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:33.041629 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.042274 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.043153 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.043413 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.043506 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:33.047144 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:43889: BlockingRecv error: recv got EOF from 127.10.225.129:43889 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":137,"threads_started":1}
Aug 06 23:52:33 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:33 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:33.259169 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:33.252917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:53727)
0806 23:52:33.253212 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:33.253226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:33.253323 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:33.253604 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:33.253606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:33.253804 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:33.253986 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:33.253998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.255093 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.255096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:33.255767 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:33.255776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.256393 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.256397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:33.256472 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:33.256989 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:33.257001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:33.258989 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":101,"threads_started":1}
W20260806 23:52:33.259307 24947 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:33.253011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:53727 (local address 127.10.225.190:40695)
0806 23:52:33.253172 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:33.253230 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:33.253275 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:52:33.253397 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:33.253399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:33.253456 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:33.253526 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:33.254127 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.254897 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.255922 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.256184 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.256296 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:33.259182 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:53727: BlockingRecv error: recv got EOF from 127.10.225.130:53727 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 23:52:33 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:33 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:33.482630 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:33.476838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:35125)
0806 23:52:33.477096 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:33.477109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:33.477207 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:33.477415 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:33.477417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:33.477593 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:33.477688 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:33.477692 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.478514 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.478516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:33.479091 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:33.479099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.479856 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.479860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:33.479956 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:33.480619 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:33.480632 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:33.482495 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
W20260806 23:52:33.482846 24947 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:33.476890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:35125 (local address 127.10.225.190:40695)
0806 23:52:33.477036 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:33.477041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:33.477173 (+ 132us) server_negotiation.cc:407] Connection header received
0806 23:52:33.477278 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:33.477280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:33.477328 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:33.477392 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:33.477865 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.478406 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.479275 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:33.479605 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:33.479795 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:33.482709 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:35125: BlockingRecv error: recv got EOF from 127.10.225.131:35125 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 23:52:34 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:34 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:34.061247 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:34.048102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:49091)
0806 23:52:34.048413 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:34.048428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:34.048551 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:34.049070 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:34.049073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:34.049296 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:34.049451 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:34.049471 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.050425 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.050428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:34.050985 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:34.050994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.051587 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.051590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:34.051658 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:34.058166 (+ 6508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:34.058181 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:34.061050 (+ 2869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":106,"threads_started":1}
W20260806 23:52:34.061477 24951 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:34.048331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:49091 (local address 127.10.225.190:40695)
0806 23:52:34.048700 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:34.048706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:34.048725 (+ 19us) server_negotiation.cc:407] Connection header received
0806 23:52:34.048797 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:34.048801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:34.048867 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:34.049027 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:34.049682 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.050321 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.051118 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.051444 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.051548 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:34.061298 (+ 9750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:49091: BlockingRecv error: recv got EOF from 127.10.225.129:49091 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":125,"threads_started":1}
Aug 06 23:52:34 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:34 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:34.266921 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:34.260263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:51021)
0806 23:52:34.260669 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:34.260689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:34.260830 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:34.261084 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:34.261087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:34.261300 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:34.261407 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:34.261413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.262150 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.262154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:34.262697 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:34.262704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.263508 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.263513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:34.263775 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:34.264439 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:34.264452 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:34.266786 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":128,"threads_started":1}
W20260806 23:52:34.267287 24951 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:34.260435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:51021 (local address 127.10.225.190:40695)
0806 23:52:34.260672 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:34.260677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:34.260771 (+ 94us) server_negotiation.cc:407] Connection header received
0806 23:52:34.260924 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:34.260928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:34.260980 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:34.261046 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:34.261520 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.262047 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.262807 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.263269 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.263457 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:34.267090 (+ 3633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:51021: BlockingRecv error: recv got EOF from 127.10.225.130:51021 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 06 23:52:34 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:34 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:34.489995 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:34.483652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:44849)
0806 23:52:34.484079 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:34.484096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:34.484230 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:34.484561 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:34.484565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:34.484800 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:34.484980 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:34.484990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.486006 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.486009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:34.486589 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:34.486598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.487125 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.487127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:34.487194 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:34.487786 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:34.487798 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:34.489824 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":123,"threads_started":1}
W20260806 23:52:34.490156 24951 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:34.483795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:44849 (local address 127.10.225.190:40695)
0806 23:52:34.483996 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:34.484002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:34.484213 (+ 211us) server_negotiation.cc:407] Connection header received
0806 23:52:34.484319 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:34.484322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:34.484389 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:34.484470 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:34.485179 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.485845 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.486736 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:34.487005 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:34.487105 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:34.490037 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:44849: BlockingRecv error: recv got EOF from 127.10.225.131:44849 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 23:52:35 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:35 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:35.068646 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:35.062132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:33247)
0806 23:52:35.062402 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:35.062416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:35.062501 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:35.063032 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:35.063034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:35.063234 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:35.063348 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:35.063354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.064312 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.064315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:35.064925 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:35.064934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.065509 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.065512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:35.065589 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:35.066154 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:35.066167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:35.068489 (+ 2322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
W20260806 23:52:35.068845 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:35.062330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:33247 (local address 127.10.225.190:40695)
0806 23:52:35.062698 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:35.062703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:35.062723 (+ 20us) server_negotiation.cc:407] Connection header received
0806 23:52:35.062783 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:35.062785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:35.062843 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:35.062971 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:35.063525 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.064188 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.065084 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.065365 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.065467 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:35.068693 (+ 3226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:33247: BlockingRecv error: recv got EOF from 127.10.225.129:33247 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":113,"threads_started":1}
Aug 06 23:52:35 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:35 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:35.274255 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:35.268436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:51127)
0806 23:52:35.268756 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:35.268773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:35.268909 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:35.269268 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:35.269271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:35.269539 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:35.269643 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:35.269650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.270542 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.270544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:35.271126 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:35.271135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.271593 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.271597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:35.271713 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:35.272252 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:35.272264 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:35.274129 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
W20260806 23:52:35.274454 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:35.268689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:51127 (local address 127.10.225.190:40695)
0806 23:52:35.268823 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:35.268828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:35.268844 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:52:35.269052 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:35.269056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:35.269114 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:35.269215 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:35.269746 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.270433 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.271242 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.271472 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.271550 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:35.274324 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:51127: BlockingRecv error: recv got EOF from 127.10.225.130:51127 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 23:52:35 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:35 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:35.496232 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:35.490834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:50911)
0806 23:52:35.491112 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:35.491124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:35.491225 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:35.491464 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:35.491467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:35.491696 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:35.491794 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:35.491799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.492550 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.492553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:35.493091 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:35.493099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.493519 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.493522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:35.493598 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:35.494087 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:35.494098 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:35.496067 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
W20260806 23:52:35.496352 24955 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:35.490903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:50911 (local address 127.10.225.190:40695)
0806 23:52:35.491047 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:35.491051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:35.491190 (+ 139us) server_negotiation.cc:407] Connection header received
0806 23:52:35.491309 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:35.491311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:35.491364 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:35.491429 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:35.491904 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.492452 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.493203 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:35.493419 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:35.493504 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:35.496234 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:50911: BlockingRecv error: recv got EOF from 127.10.225.131:50911 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 23:52:36 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:36 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:36.075068 24961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:36.069450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:43447)
0806 23:52:36.069725 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:36.069738 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:36.069838 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:36.070046 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:36.070048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:36.070232 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:36.070328 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:36.070333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.071157 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.071160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:36.071787 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:36.071796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.072388 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.072391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:36.072461 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:36.072952 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:36.072962 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:36.074931 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
W20260806 23:52:36.075264 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:36.069529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:43447 (local address 127.10.225.190:40695)
0806 23:52:36.069808 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:36.069811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:36.069824 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:52:36.069915 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:36.069917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:36.069962 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:36.070035 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:36.070450 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.070998 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.071996 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.072231 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.072324 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:36.075109 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:43447: BlockingRecv error: recv got EOF from 127.10.225.129:43447 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
Aug 06 23:52:36 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:36 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:36.281658 24963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:36.275324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:52341)
0806 23:52:36.275609 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:36.275623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:36.275759 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:36.275993 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:36.275995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:36.276191 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:36.276290 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:36.276294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.277096 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.277099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:36.277671 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:36.277680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.278152 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.278155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:36.278224 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:36.278889 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:36.278909 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:36.281452 (+ 2543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
W20260806 23:52:36.281759 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:36.275501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:52341 (local address 127.10.225.190:40695)
0806 23:52:36.275699 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:36.275703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:36.275718 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:52:36.275845 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:36.275847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:36.275901 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:36.275968 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:36.276425 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.276972 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.277799 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.278023 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.278103 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:36.281645 (+ 3542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:52341: BlockingRecv error: recv got EOF from 127.10.225.130:52341 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 06 23:52:36 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:36 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:36.503152 24965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:36.497219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:37159)
0806 23:52:36.497553 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:36.497570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:36.497772 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:36.498035 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:36.498038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:36.498242 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:36.498344 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:36.498349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.499074 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.499077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:36.499973 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:36.499985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.500458 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.500462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:36.500545 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:36.501094 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:36.501108 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:36.503028 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
W20260806 23:52:36.503331 24962 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:36.497372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:37159 (local address 127.10.225.190:40695)
0806 23:52:36.497554 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:36.497558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:36.497722 (+ 164us) server_negotiation.cc:407] Connection header received
0806 23:52:36.497878 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:36.497881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:36.497931 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:36.497998 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:36.498456 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.498974 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.500093 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:36.500331 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:36.500426 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:36.503189 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:37159: BlockingRecv error: recv got EOF from 127.10.225.131:37159 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 23:52:37 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:37 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:37.081965 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:37.076094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:56163)
0806 23:52:37.076685 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:37.076700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:37.076833 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:37.077092 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:37.077095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:37.077335 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:37.077435 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:37.077441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.078194 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.078198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:37.078746 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:37.078754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.079175 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.079178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:37.079254 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:37.079885 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:37.079901 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:37.081840 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":93,"threads_started":1}
W20260806 23:52:37.082126 24967 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:37.076288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:56163 (local address 127.10.225.190:40695)
0806 23:52:37.076549 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:37.076554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:37.076801 (+ 247us) server_negotiation.cc:407] Connection header received
0806 23:52:37.076928 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:37.076931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:37.076982 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:37.077058 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:37.077552 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.078092 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.078859 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.079075 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.079159 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:37.081982 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:56163: BlockingRecv error: recv got EOF from 127.10.225.129:56163 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":65,"threads_started":1}
Aug 06 23:52:37 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:37 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:37.288210 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:37.282459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:60693)
0806 23:52:37.282785 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:37.282798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:37.282895 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:37.283153 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:37.283156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:37.283357 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:37.283459 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:37.283465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.284299 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.284302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:37.284914 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:37.284924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.285543 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.285547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:37.285614 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:37.286158 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:37.286170 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:37.288051 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":77,"threads_started":1}
W20260806 23:52:37.288302 24967 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:37.282542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:60693 (local address 127.10.225.190:40695)
0806 23:52:37.282679 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:37.282682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:37.282855 (+ 173us) server_negotiation.cc:407] Connection header received
0806 23:52:37.282962 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:37.282964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:37.283016 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:37.283082 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:37.283569 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.284190 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.285032 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.285243 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.285585 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:37.288199 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:60693: BlockingRecv error: recv got EOF from 127.10.225.130:60693 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 23:52:37 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:37 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:37.509610 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:37.504048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:43633)
0806 23:52:37.504437 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:37.504453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:37.504568 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:37.504806 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:37.504808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:37.505020 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:37.505116 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:37.505121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.505855 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.505859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:37.506406 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:37.506414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.506824 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.506827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:37.506940 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:37.507461 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:37.507473 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:37.509394 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":104,"threads_started":1}
W20260806 23:52:37.509663 24967 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:37.504128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:43633 (local address 127.10.225.190:40695)
0806 23:52:37.504264 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:37.504269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:37.504520 (+ 251us) server_negotiation.cc:407] Connection header received
0806 23:52:37.504657 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:37.504660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:37.504709 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:37.504774 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:37.505227 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.505747 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.506518 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:37.506725 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:37.506808 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:37.509558 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:43633: BlockingRecv error: recv got EOF from 127.10.225.131:43633 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 23:52:38 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:38 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:38.089753 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:38.083165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:34029)
0806 23:52:38.083558 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:38.083574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:38.083757 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:38.084178 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:38.084182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:38.084423 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:38.084553 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:38.084560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.085558 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.085562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:38.086551 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:38.086564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.087198 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.087203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:38.087288 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:38.087842 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:38.087854 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:38.089628 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
W20260806 23:52:38.089941 24974 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:38.083455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:34029 (local address 127.10.225.190:40695)
0806 23:52:38.083731 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:38.083735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:38.083754 (+ 19us) server_negotiation.cc:407] Connection header received
0806 23:52:38.083917 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:38.083920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:38.083977 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:38.084074 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:38.084695 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.085393 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.086658 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.086918 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.087021 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:38.089780 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:34029: BlockingRecv error: recv got EOF from 127.10.225.129:34029 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
Aug 06 23:52:38 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:38 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:38.295833 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:38.289301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:46687)
0806 23:52:38.289723 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:38.289739 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:38.289886 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:38.290292 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:38.290297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:38.290499 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:38.290608 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:38.290614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.291527 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.291530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:38.292198 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:38.292210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.292763 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.292767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:38.292848 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:38.293441 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:38.293459 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:38.295631 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":215,"threads_started":1}
W20260806 23:52:38.295944 24974 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:38.289453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:46687 (local address 127.10.225.190:40695)
0806 23:52:38.289760 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:38.289766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:38.289814 (+ 48us) server_negotiation.cc:407] Connection header received
0806 23:52:38.290017 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:38.290020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:38.290077 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:38.290177 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:38.290738 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.291423 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.292326 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.292646 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.292755 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:38.295842 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:46687: BlockingRecv error: recv got EOF from 127.10.225.130:46687 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 06 23:52:38 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:38 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:38.515776 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:38.510479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:39363)
0806 23:52:38.510776 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:38.510789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:38.510886 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:38.511109 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:38.511111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:38.511291 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:38.511385 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:38.511390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.512182 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.512185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:38.512741 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:38.512749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.513188 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.513190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:38.513247 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:38.513771 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:38.513783 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:38.515611 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
W20260806 23:52:38.515929 24974 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:38.510568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:39363 (local address 127.10.225.190:40695)
0806 23:52:38.510695 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:38.510698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:38.510848 (+ 150us) server_negotiation.cc:407] Connection header received
0806 23:52:38.510967 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:38.510969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:38.511015 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:38.511087 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:38.511515 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.512078 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.512871 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:38.513092 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:38.513171 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:38.515815 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:39363: BlockingRecv error: recv got EOF from 127.10.225.131:39363 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 23:52:39 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:39 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:39.099272 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:39.090591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:34081)
0806 23:52:39.090875 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:39.090892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:39.091003 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:39.091250 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:39.091252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:39.091437 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:39.091535 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:39.091540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.092422 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.092426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:39.093232 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:39.093244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.093776 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.093779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:39.093847 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:39.097243 (+ 3396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:39.097256 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:39.099134 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
W20260806 23:52:39.099434 24980 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:39.090747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:34081 (local address 127.10.225.190:40695)
0806 23:52:39.091034 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:39.091038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:39.091052 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:39.091094 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:39.091096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:39.091145 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:39.091227 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:39.091652 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.092226 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.093378 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.093641 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.093743 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:39.099312 (+ 5569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:34081: BlockingRecv error: recv got EOF from 127.10.225.129:34081 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
Aug 06 23:52:39 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:39 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:39.303593 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:39.296817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:37703)
0806 23:52:39.297080 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:39.297093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:39.297197 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:39.297443 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:39.297446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:39.297645 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:39.297742 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:39.297748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.298495 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.298497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:39.299071 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:39.299079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.299877 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.299882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:39.299983 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:39.300726 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:39.300739 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:39.303325 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
W20260806 23:52:39.303932 24980 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:39.296907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:37703 (local address 127.10.225.190:40695)
0806 23:52:39.297064 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:39.297068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:39.297156 (+ 88us) server_negotiation.cc:407] Connection header received
0806 23:52:39.297268 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:39.297270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:39.297324 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:39.297393 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:39.297878 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.298395 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.299275 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.299699 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.299814 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:39.303788 (+ 3974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:37703: BlockingRecv error: recv got EOF from 127.10.225.130:37703 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 23:52:39 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:39 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:39.522535 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:39.516651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:54765)
0806 23:52:39.516986 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:39.517004 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:39.517124 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:39.517382 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:39.517385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:39.517592 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:39.517690 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:39.517696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.518457 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.518460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:39.519006 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:39.519014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.519445 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.519456 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:39.519536 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:39.520107 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:39.520119 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:39.522358 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":96,"threads_started":1}
W20260806 23:52:39.522625 24980 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:39.516789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:54765 (local address 127.10.225.190:40695)
0806 23:52:39.517052 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:39.517057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:39.517071 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:39.517224 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:39.517228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:39.517278 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:39.517344 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:39.517809 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.518344 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.519121 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:39.519343 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:39.519428 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:39.522509 (+ 3081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:54765: BlockingRecv error: recv got EOF from 127.10.225.131:54765 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Aug 06 23:52:40 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:40 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:40.106225 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:40.100483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:58909)
0806 23:52:40.100828 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:40.100844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:40.100969 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:40.101347 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:40.101351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:40.101566 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:40.101677 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:40.101683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.102460 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.102463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:40.103025 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:40.103033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.103449 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.103452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:40.103516 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:40.104086 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:40.104100 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:40.106101 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
W20260806 23:52:40.106371 24988 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:40.100621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:58909 (local address 127.10.225.190:40695)
0806 23:52:40.100924 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:40.100928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:40.100957 (+ 29us) server_negotiation.cc:407] Connection header received
0806 23:52:40.101068 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:40.101071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:40.101117 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:40.101224 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:40.101809 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.102350 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.103122 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.103340 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.103431 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:40.106251 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:58909: BlockingRecv error: recv got EOF from 127.10.225.129:58909 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
Aug 06 23:52:40 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:40 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:40.310274 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:40.304606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:40163)
0806 23:52:40.304870 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:40.304883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:40.304975 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:40.305192 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:40.305194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:40.305379 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:40.305478 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:40.305483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.306350 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.306353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:40.306919 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:40.306936 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.307461 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.307464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:40.307546 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:40.308173 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:40.308186 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:40.310115 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
W20260806 23:52:40.310376 24988 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:40.304702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:40163 (local address 127.10.225.190:40695)
0806 23:52:40.304872 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:40.304876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:40.304947 (+ 71us) server_negotiation.cc:407] Connection header received
0806 23:52:40.305050 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:40.305052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:40.305097 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:40.305169 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:40.305625 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.306174 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.307052 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.307337 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.307443 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:40.310263 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:40163: BlockingRecv error: recv got EOF from 127.10.225.130:40163 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 23:52:40 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:40 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:40.529096 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:40.523603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:56153)
0806 23:52:40.523947 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:40.523962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:40.524053 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:40.524323 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:40.524327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:40.524559 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:40.524710 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:40.524719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.525536 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.525539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:40.526109 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:40.526117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.526549 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.526552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:40.526629 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:40.527109 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:40.527123 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:40.528941 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
W20260806 23:52:40.529186 24988 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:40.523853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:56153 (local address 127.10.225.190:40695)
0806 23:52:40.523984 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:40.523988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:40.524019 (+ 31us) server_negotiation.cc:407] Connection header received
0806 23:52:40.524149 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:40.524152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:40.524201 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:40.524286 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:40.524840 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.525429 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.526220 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:40.526447 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:40.526532 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:40.529081 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:56153: BlockingRecv error: recv got EOF from 127.10.225.131:56153 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 23:52:41 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:41 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:41.113070 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:41.107146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:59279)
0806 23:52:41.107530 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:41.107546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:41.107705 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:41.108205 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:41.108208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:41.108446 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:41.108589 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:41.108598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.109451 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.109454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:41.110043 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:41.110051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.110535 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.110538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:41.110601 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:41.111094 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:41.111105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:41.112939 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":175,"threads_started":1}
W20260806 23:52:41.113196 24993 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:41.107235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:59279 (local address 127.10.225.190:40695)
0806 23:52:41.107530 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:41.107535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:41.107608 (+ 73us) server_negotiation.cc:407] Connection header received
0806 23:52:41.107857 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:41.107861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:41.107954 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:41.108157 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:41.108712 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.109258 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.110182 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.110417 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.110507 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:41.113078 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:59279: BlockingRecv error: recv got EOF from 127.10.225.129:59279 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":77,"threads_started":1}
Aug 06 23:52:41 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:41 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:41.316932 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:41.311247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:49233)
0806 23:52:41.311577 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:41.311592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:41.311752 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:41.312016 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:41.312019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:41.312223 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:41.312319 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:41.312325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.313058 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.313061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:41.313603 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:41.313611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.314035 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.314039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:41.314116 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:41.314610 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:41.314622 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:41.316809 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":95,"threads_started":1}
W20260806 23:52:41.317117 24993 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:41.311401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:49233 (local address 127.10.225.190:40695)
0806 23:52:41.311579 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:41.311585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:41.311664 (+ 79us) server_negotiation.cc:407] Connection header received
0806 23:52:41.311853 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:41.311856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:41.311909 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:41.311977 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:41.312431 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.312956 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.313711 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.313934 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.314018 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:41.316997 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:49233: BlockingRecv error: recv got EOF from 127.10.225.130:49233 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 23:52:41 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:41 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:41.536375 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:41.530171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:44997)
0806 23:52:41.530527 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:41.530544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:41.530667 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:41.530935 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:41.530938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:41.531141 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:41.531237 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:41.531243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.532123 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.532127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:41.532890 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:41.532902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.533509 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.533512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:41.533586 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:41.534095 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:41.534107 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:41.536157 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":191,"threads_started":1}
W20260806 23:52:41.536460 24993 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:41.530433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:44997 (local address 127.10.225.190:40695)
0806 23:52:41.530624 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:41.530629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:41.530646 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:52:41.530769 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:41.530772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:41.530823 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:41.530895 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:41.531354 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.532009 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.533071 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:41.533361 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:41.533469 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:41.536340 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:44997: BlockingRecv error: recv got EOF from 127.10.225.131:44997 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 23:52:42 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:42 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:42.123057 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:42.114178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:35585)
0806 23:52:42.114482 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:42.114495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:42.114588 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:42.114882 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:42.114885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:42.115076 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:42.115178 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:42.115184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.116060 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.116064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:42.116722 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:42.116732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.117262 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.117266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:42.117342 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:42.120461 (+ 3119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:42.120474 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:42.122862 (+ 2388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
W20260806 23:52:42.123231 24998 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:42.114301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:35585 (local address 127.10.225.190:40695)
0806 23:52:42.114548 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:42.114552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:42.114567 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:52:42.114670 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:42.114672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:42.114718 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:42.114809 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:42.115274 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.115933 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.116844 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.117148 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.117245 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:42.123079 (+ 5834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:35585: BlockingRecv error: recv got EOF from 127.10.225.129:35585 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
Aug 06 23:52:42 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:42 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:42.323215 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:42.317820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:58269)
0806 23:52:42.318184 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:42.318203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:42.318326 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:42.318573 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:42.318575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:42.318787 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:42.318898 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:42.318904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.319660 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.319663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:42.320250 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:42.320259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.320692 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.320695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:42.320770 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:42.321290 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:42.321301 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:42.323054 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":130,"threads_started":1}
W20260806 23:52:42.323309 24998 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:42.317973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:58269 (local address 127.10.225.190:40695)
0806 23:52:42.318163 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:42.318168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:42.318278 (+ 110us) server_negotiation.cc:407] Connection header received
0806 23:52:42.318421 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:42.318424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:42.318469 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:42.318537 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:42.319021 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.319555 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.320376 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.320588 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.320673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:42.323203 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:58269: BlockingRecv error: recv got EOF from 127.10.225.130:58269 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 23:52:42 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:42 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:42.543987 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:42.537276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:44353)
0806 23:52:42.537627 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:42.537639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:42.537787 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:42.538144 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:42.538147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:42.538343 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:42.538461 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:42.538469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.539750 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.539756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:42.540469 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:42.540487 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.541415 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.541419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:42.541516 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:42.542035 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:42.542047 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:42.543832 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":82,"threads_started":1}
W20260806 23:52:42.544183 24998 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:42.537353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:44353 (local address 127.10.225.190:40695)
0806 23:52:42.537576 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:42.537581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:42.537704 (+ 123us) server_negotiation.cc:407] Connection header received
0806 23:52:42.537884 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:42.537887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:42.537935 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:42.538009 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:42.538566 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.539465 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.540651 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:42.541268 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:42.541397 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:42.544036 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:44353: BlockingRecv error: recv got EOF from 127.10.225.131:44353 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 06 23:52:43 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:43 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:43.129902 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:43.124046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:38641)
0806 23:52:43.124362 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:43.124376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:43.124470 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:43.124719 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:43.124721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:43.124913 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:43.125022 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:43.125028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.125774 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.125777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:43.126353 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:43.126361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.126817 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.126820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:43.126881 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:43.127397 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:43.127409 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:43.129730 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":87,"threads_started":1}
W20260806 23:52:43.130013 25004 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:43.124153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:38641 (local address 127.10.225.190:40695)
0806 23:52:43.124467 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:43.124471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:43.124484 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:52:43.124545 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:43.124547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:43.124589 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:43.124666 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:43.125126 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.125672 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.126479 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.126706 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.126791 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:43.129866 (+ 3075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:38641: BlockingRecv error: recv got EOF from 127.10.225.129:38641 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
Aug 06 23:52:43 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:43 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:43.329959 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:43.324267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:49915)
0806 23:52:43.324574 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:43.324588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:43.324688 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:43.324944 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:43.324946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:43.325137 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:43.325241 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:43.325247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.326034 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.326037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:43.326614 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:43.326622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.327102 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.327106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:43.327175 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:43.327752 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:43.327764 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:43.329812 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
W20260806 23:52:43.330118 25004 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:43.324468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:49915 (local address 127.10.225.190:40695)
0806 23:52:43.324601 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:43.324605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:43.324638 (+ 33us) server_negotiation.cc:407] Connection header received
0806 23:52:43.324771 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:43.324774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:43.324818 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:43.324877 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:43.325379 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.325913 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.326733 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.326974 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.327056 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:43.329998 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:49915: BlockingRecv error: recv got EOF from 127.10.225.130:49915 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 23:52:43 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:43 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:43.550514 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:43.544833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:37247)
0806 23:52:43.545109 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:43.545122 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:43.545224 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:43.545568 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:43.545570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:43.545751 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:43.545853 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:43.545858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.546599 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.546602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:43.547171 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:43.547179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.547619 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.547622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:43.547708 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:43.548231 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:43.548243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:43.550355 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
W20260806 23:52:43.550611 25004 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:43.544973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:37247 (local address 127.10.225.190:40695)
0806 23:52:43.545103 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:43.545107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:43.545187 (+ 80us) server_negotiation.cc:407] Connection header received
0806 23:52:43.545374 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:43.545377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:43.545443 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:43.545523 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:43.545966 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.546501 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.547286 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:43.547512 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:43.547591 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:43.550509 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:37247: BlockingRecv error: recv got EOF from 127.10.225.131:37247 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 23:52:44 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:44 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:44.141795 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:44.130991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:38055)
0806 23:52:44.131370 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:44.131393 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:44.131487 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:44.131750 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:44.131753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:44.131953 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:44.132055 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:44.132060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.132791 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.132795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:44.133518 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:44.133530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.134019 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.134023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:44.134090 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:44.139352 (+ 5262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:44.139368 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:44.141594 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
W20260806 23:52:44.142094 25008 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:44.131121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:38055 (local address 127.10.225.190:40695)
0806 23:52:44.131498 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:44.131501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:44.131515 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:52:44.131553 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:44.131555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:44.131600 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:44.131700 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:44.132153 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.132681 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.133656 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.133889 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.133975 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:44.141802 (+ 7827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:38055: BlockingRecv error: recv got EOF from 127.10.225.129:38055 (error 108)
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":137,"threads_started":1}
Aug 06 23:52:44 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:44 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:44.337031 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:44.330996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:56187)
0806 23:52:44.331345 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:44.331361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:44.331504 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:44.331776 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:44.331779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:44.332007 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:44.332107 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:44.332114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.332883 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.332886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:44.333538 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:44.333550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.334168 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.334171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:44.334262 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:44.334849 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:44.334860 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:44.336881 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":124,"threads_started":1}
W20260806 23:52:44.337347 25008 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:44.331146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:56187 (local address 127.10.225.190:40695)
0806 23:52:44.331348 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:44.331353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:44.331441 (+ 88us) server_negotiation.cc:407] Connection header received
0806 23:52:44.331600 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:44.331603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:44.331649 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:44.331753 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:44.332225 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.332777 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.333700 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.334012 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.334122 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:44.337180 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:56187: BlockingRecv error: recv got EOF from 127.10.225.130:56187 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 23:52:44 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:44 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.131@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:44.557304 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:44.551481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.131:60177)
0806 23:52:44.551836 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:44.551853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:44.551984 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:44.552292 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:44.552294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:44.552529 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:44.552652 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:44.552658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.553567 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.553570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:44.554157 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:44.554166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.554602 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.554605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:44.554671 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:44.555236 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:44.555247 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:44.557144 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
W20260806 23:52:44.557392 25008 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:44.551649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:60177 (local address 127.10.225.190:40695)
0806 23:52:44.551837 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:44.551842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:44.551921 (+ 79us) server_negotiation.cc:407] Connection header received
0806 23:52:44.552102 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:44.552104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:44.552160 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:44.552243 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:44.552762 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.553451 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.554269 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:44.554485 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:44.554560 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:44.557267 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.131:60177: BlockingRecv error: recv got EOF from 127.10.225.131:60177 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 23:52:45 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:45 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.129@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:45.150132 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:45.143229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.129:45631)
0806 23:52:45.143762 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:45.143780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:45.143909 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:45.144289 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:45.144292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:45.144713 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:45.144862 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:45.144874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.145887 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.145890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:45.146465 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:45.146475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.147070 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.147073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:45.147143 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:45.147723 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:45.147735 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:45.150003 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":121,"threads_started":1}
W20260806 23:52:45.150326 25014 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:45.143389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:45631 (local address 127.10.225.190:40695)
0806 23:52:45.143843 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:45.143848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:45.143868 (+ 20us) server_negotiation.cc:407] Connection header received
0806 23:52:45.144025 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:45.144029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:45.144092 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:45.144220 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:45.145013 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.145751 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.146638 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.146938 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.147046 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:45.150186 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.129:45631: BlockingRecv error: recv got EOF from 127.10.225.129:45631 (error 108)
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":165,"threads_started":1}
Aug 06 23:52:45 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 23:52:45 dist-test-slave-nj21 krb5kdc[23564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.225.130@KRBTEST.COM for oryx/127.10.225.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 23:52:45.344301 25015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:45.338061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:40695 (local address 127.10.225.130:57427)
0806 23:52:45.338351 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:45.338368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:45.338488 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:45.338761 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:45.338763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:45.338996 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:45.339095 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:45.339101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.339902 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.339906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:45.340494 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:45.340504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.340992 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.340996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:45.341106 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:45.341798 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:45.341810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:45.344167 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.225.190:40695: Server oryx/127.10.225.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
W20260806 23:52:45.344609 25014 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:45.338211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:57427 (local address 127.10.225.190:40695)
0806 23:52:45.338354 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:45.338359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:45.338443 (+ 84us) server_negotiation.cc:407] Connection header received
0806 23:52:45.338578 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:45.338581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:45.338635 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:45.338720 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:45.339208 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.339807 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.340613 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.340864 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.340958 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:45.344389 (+ 3431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.225.130:57427: BlockingRecv error: recv got EOF from 127.10.225.130:57427 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:52:45.446581 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 24356
I20260806 23:52:45.479583 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 24562
I20260806 23:52:45.493381 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 24745
I20260806 23:52:45.507313 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 24272
2026-08-06T23:52:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18450 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T23:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:45Z Disabled control of system clock
I20260806 23:52:45.545509 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:37001
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:37001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 23:52:45.668694 25020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:45.669010 25020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:45.669063 25020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:45.673507 25020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:45.673583 25020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:45.673604 25020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:45.673624 25020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:45.673653 25020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:45.680459 25020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:37001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:37001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:45.681703 25020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:45.683132 25020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:45.692569 25027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:45.692557 25031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:45.692553 25028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:45.692740 25020 server_base.cc:1034] running on GCE node
I20260806 23:52:45.693416 25020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:45.694561 25020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:45.695753 25020 hybrid_clock.cc:648] HybridClock initialized: now 1786060365695740 us; error 28 us; skew 500 ppm
I20260806 23:52:45.697949 25020 webserver.cc:460] Webserver started at http://127.10.225.190:45835/ using document root <none> and password file <none>
I20260806 23:52:45.698571 25020 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:45.698642 25020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:45.698848 25020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:45.701757 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "1a7bd57e4ffa479cb78061ecdf91f92f"
format_stamp: "Formatted at 2026-08-06 23:52:45 on dist-test-slave-nj21"
server_key: "f6c40e03bdf356aa2b95cc0e6b0700ef"
server_key_iv: "37af517043f4d9227aef19130f9e7ed0"
server_key_version: "encryptionkey@0"
I20260806 23:52:45.702512 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "1a7bd57e4ffa479cb78061ecdf91f92f"
format_stamp: "Formatted at 2026-08-06 23:52:45 on dist-test-slave-nj21"
server_key: "f6c40e03bdf356aa2b95cc0e6b0700ef"
server_key_iv: "37af517043f4d9227aef19130f9e7ed0"
server_key_version: "encryptionkey@0"
I20260806 23:52:45.708250 25020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 23:52:45.711195 25039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:45.712598 25020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:45.712716 25020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "1a7bd57e4ffa479cb78061ecdf91f92f"
format_stamp: "Formatted at 2026-08-06 23:52:45 on dist-test-slave-nj21"
server_key: "f6c40e03bdf356aa2b95cc0e6b0700ef"
server_key_iv: "37af517043f4d9227aef19130f9e7ed0"
server_key_version: "encryptionkey@0"
I20260806 23:52:45.712841 25020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:45.726018 25020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:45.726754 25020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:45.726948 25020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:45.736142 25020 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:37001
I20260806 23:52:45.736158 25114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:37001 every 8 connection(s)
I20260806 23:52:45.737155 25020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:45.740610 25115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:45.743985 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 25020
I20260806 23:52:45.744088 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:45.744375 11142 external_mini_cluster.cc:1428] Setting key dcee242997d97c8001bfe624412d2ac5
I20260806 23:52:45.746927 25115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f: Bootstrap starting.
I20260806 23:52:45.749271 25115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:45.750032 25115 log.cc:824] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:45.752776 25115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f: No bootstrap required, opened a new log
I20260806 23:52:45.754350 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:45.745861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.10.225.190:37001)
0806 23:52:45.746519 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:45.746535 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:45.746572 (+ 37us) server_negotiation.cc:407] Connection header received
0806 23:52:45.747235 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:45.747253 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:45.747651 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:45.748000 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:45.748437 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.749754 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.750834 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:45.751324 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:45.751490 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:45.752289 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:45.752295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:45.752308 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:45.752709 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:45.752769 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:45.752790 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:45.752882 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:45.753023 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 23:52:45.753324 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260806 23:52:45.758008 25115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7bd57e4ffa479cb78061ecdf91f92f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37001 } }
I20260806 23:52:45.758251 25115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:45.758322 25115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a7bd57e4ffa479cb78061ecdf91f92f, State: Initialized, Role: FOLLOWER
I20260806 23:52:45.758759 25115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7bd57e4ffa479cb78061ecdf91f92f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37001 } }
I20260806 23:52:45.758890 25115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:45.758955 25115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:45.759052 25115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:45.760145 25115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7bd57e4ffa479cb78061ecdf91f92f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37001 } }
I20260806 23:52:45.760489 25115 leader_election.cc:302] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a7bd57e4ffa479cb78061ecdf91f92f; no voters:
I20260806 23:52:45.760808 25115 leader_election.cc:288] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:45.760910 25124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:45.761145 25124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [term 1 LEADER]: Becoming Leader. State: Replica: 1a7bd57e4ffa479cb78061ecdf91f92f, State: Running, Role: LEADER
I20260806 23:52:45.761516 25124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7bd57e4ffa479cb78061ecdf91f92f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37001 } }
I20260806 23:52:45.762002 25115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:45.763054 25125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a7bd57e4ffa479cb78061ecdf91f92f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7bd57e4ffa479cb78061ecdf91f92f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37001 } } }
I20260806 23:52:45.763180 25125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:45.763547 25134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:45.764096 25126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a7bd57e4ffa479cb78061ecdf91f92f. Latest consensus state: current_term: 1 leader_uuid: "1a7bd57e4ffa479cb78061ecdf91f92f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7bd57e4ffa479cb78061ecdf91f92f" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37001 } } }
I20260806 23:52:45.764225 25126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:45.766911 25134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:45.772943 25134 catalog_manager.cc:1347] Generated new cluster ID: cb77f520d10c464a80845118bd11450a
I20260806 23:52:45.773020 25134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:45.791196 25134 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:45.792239 25134 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:45.797669 25134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a7bd57e4ffa479cb78061ecdf91f92f: Generated new TSK 0
I20260806 23:52:45.798518 25134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:52:45.803249 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37001
--builtin_ntp_servers=127.10.225.148:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 23:52:45.926396 25148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:45.926678 25148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:45.926716 25148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:45.930763 25148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:45.930828 25148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:45.930893 25148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:45.936079 25148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:45.937197 25148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:45.938606 25148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:45.948777 25154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:45.948840 25155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:45.948777 25158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:45.949222 25148 server_base.cc:1034] running on GCE node
I20260806 23:52:45.949615 25148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:45.950238 25148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:45.951402 25148 hybrid_clock.cc:648] HybridClock initialized: now 1786060365951377 us; error 42 us; skew 500 ppm
I20260806 23:52:45.953326 25148 webserver.cc:460] Webserver started at http://127.10.225.129:45097/ using document root <none> and password file <none>
I20260806 23:52:45.953924 25148 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:45.953994 25148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:45.954195 25148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:45.956368 25148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "43b39dcd630e47db8974ca041a121cab"
format_stamp: "Formatted at 2026-08-06 23:52:45 on dist-test-slave-nj21"
server_key: "29d0c7d3c22ac8f5a8abea26b25cd428"
server_key_iv: "8cb28afc0732897b4e414185352ecae6"
server_key_version: "encryptionkey@0"
I20260806 23:52:45.956925 25148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "43b39dcd630e47db8974ca041a121cab"
format_stamp: "Formatted at 2026-08-06 23:52:45 on dist-test-slave-nj21"
server_key: "29d0c7d3c22ac8f5a8abea26b25cd428"
server_key_iv: "8cb28afc0732897b4e414185352ecae6"
server_key_version: "encryptionkey@0"
I20260806 23:52:45.960695 25148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:52:45.963665 25165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:45.965175 25148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:45.965302 25148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "43b39dcd630e47db8974ca041a121cab"
format_stamp: "Formatted at 2026-08-06 23:52:45 on dist-test-slave-nj21"
server_key: "29d0c7d3c22ac8f5a8abea26b25cd428"
server_key_iv: "8cb28afc0732897b4e414185352ecae6"
server_key_version: "encryptionkey@0"
I20260806 23:52:45.965413 25148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:45.991775 25148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:45.992573 25148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:45.992754 25148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:45.993399 25148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:45.994505 25148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:45.994556 25148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:45.994613 25148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:45.994642 25148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.003886 25148 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:36983
I20260806 23:52:46.003939 25317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:36983 every 8 connection(s)
I20260806 23:52:46.004799 25148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:46.013023 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 25148
I20260806 23:52:46.013161 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:46.013461 11142 external_mini_cluster.cc:1428] Setting key 03faedf9e800e2df8281c00c9876fe02
I20260806 23:52:46.014977 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:46.007066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:40795 (local address 127.10.225.190:37001)
0806 23:52:46.007207 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:46.007212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:46.008401 (+ 1189us) server_negotiation.cc:407] Connection header received
0806 23:52:46.009920 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:46.009923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:46.009977 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:46.010054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:46.010934 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.011478 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.012693 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.013008 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.013121 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:46.014510 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:46.014514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:46.014516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:46.014529 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:46.014549 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:46.014556 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:46.014617 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:46.014763 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 23:52:46.014831 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:52:46.015726 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37001
--builtin_ntp_servers=127.10.225.148:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 23:52:46.016021 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:46.007482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:37001 (local address 127.10.225.129:40795)
0806 23:52:46.008182 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:46.008222 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:46.009712 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:46.010199 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:46.010206 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:46.010283 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:46.010784 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:46.010796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.011598 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.011601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:46.012543 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:46.012554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.013187 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.013191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:46.013370 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:46.014101 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:46.014103 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:46.014265 (+ 162us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:46.014358 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:46.014629 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:46.014638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:46.014645 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 23:52:46.014767 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 23:52:46.015072 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":145,"threads_started":1}
I20260806 23:52:46.017457 25318 heartbeater.cc:347] Connected to a master server at 127.10.225.190:37001
I20260806 23:52:46.017699 25318 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:46.018240 25318 heartbeater.cc:511] Master 127.10.225.190:37001 requested a full tablet report, sending...
I20260806 23:52:46.019903 25067 ts_manager.cc:194] Registered new tserver with Master: 43b39dcd630e47db8974ca041a121cab (127.10.225.129:36983)
I20260806 23:52:46.021552 25067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.129:40795
W20260806 23:52:46.139281 25321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:46.139555 25321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:46.139601 25321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:46.144049 25321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:46.144119 25321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:46.144177 25321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:46.149230 25321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:46.150377 25321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:46.151834 25321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:46.162395 25328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:46.162460 25329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:46.162559 25321 server_base.cc:1034] running on GCE node
W20260806 23:52:46.162395 25332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:46.163261 25321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:46.163985 25321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:46.165164 25321 hybrid_clock.cc:648] HybridClock initialized: now 1786060366165143 us; error 34 us; skew 500 ppm
I20260806 23:52:46.167146 25321 webserver.cc:460] Webserver started at http://127.10.225.130:33621/ using document root <none> and password file <none>
I20260806 23:52:46.167781 25321 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:46.167860 25321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:46.168063 25321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:46.170424 25321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "c8f121c4a5c64ac3aee0171b5d81847b"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "e3ffedd138186e94ce2b665d0142776f"
server_key_iv: "8feab1e1fae474f34576083eac79887a"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.171037 25321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "c8f121c4a5c64ac3aee0171b5d81847b"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "e3ffedd138186e94ce2b665d0142776f"
server_key_iv: "8feab1e1fae474f34576083eac79887a"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.174961 25321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:46.177677 25338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.179229 25321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:46.179366 25321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "c8f121c4a5c64ac3aee0171b5d81847b"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "e3ffedd138186e94ce2b665d0142776f"
server_key_iv: "8feab1e1fae474f34576083eac79887a"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.179481 25321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:46.189419 25321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:46.190172 25321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:46.190362 25321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:46.190974 25321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:46.192009 25321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:46.192060 25321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.192116 25321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:46.192142 25321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.201960 25321 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:33827
I20260806 23:52:46.201979 25491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:33827 every 8 connection(s)
I20260806 23:52:46.202893 25321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:46.204238 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 25321
I20260806 23:52:46.204355 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:46.204641 11142 external_mini_cluster.cc:1428] Setting key c9d5c7fb123244bee4014c772b685d45
I20260806 23:52:46.207291 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37001
--builtin_ntp_servers=127.10.225.148:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 23:52:46.214802 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:46.204921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:44733 (local address 127.10.225.190:37001)
0806 23:52:46.205082 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:46.205086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:46.206816 (+ 1730us) server_negotiation.cc:407] Connection header received
0806 23:52:46.208817 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:46.208820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:46.208871 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:46.208946 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:46.210596 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.211333 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.212627 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.212965 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.213078 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:46.214232 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:46.214236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:46.214239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:46.214258 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:46.214283 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:46.214294 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:46.214374 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:46.214551 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 23:52:46.214623 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 23:52:46.215663 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:46.205400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:37001 (local address 127.10.225.130:44733)
0806 23:52:46.206560 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:46.206648 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:46.208548 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:46.209234 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:46.209256 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:46.209431 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:46.210271 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:46.210352 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.211474 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.211477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:46.212475 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:46.212489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.213111 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.213114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:46.213269 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:46.213938 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:46.213939 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:46.213983 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:46.214059 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:46.214399 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:46.214408 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:46.214414 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:52:46.214528 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 23:52:46.214783 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":149,"threads_started":1}
I20260806 23:52:46.217062 25492 heartbeater.cc:347] Connected to a master server at 127.10.225.190:37001
I20260806 23:52:46.217337 25492 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:46.217945 25492 heartbeater.cc:511] Master 127.10.225.190:37001 requested a full tablet report, sending...
I20260806 23:52:46.219051 25067 ts_manager.cc:194] Registered new tserver with Master: c8f121c4a5c64ac3aee0171b5d81847b (127.10.225.130:33827)
I20260806 23:52:46.219884 25067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.130:44733
W20260806 23:52:46.334322 25495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:46.334627 25495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:46.334669 25495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:46.339828 25495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:46.339895 25495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:46.339970 25495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:46.345695 25495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:46.346832 25495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:46.348264 25495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:46.358342 25507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:46.358382 25505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:46.358356 25504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:46.358726 25495 server_base.cc:1034] running on GCE node
I20260806 23:52:46.359133 25495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:46.359864 25495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:46.361061 25495 hybrid_clock.cc:648] HybridClock initialized: now 1786060366361034 us; error 50 us; skew 500 ppm
I20260806 23:52:46.363040 25495 webserver.cc:460] Webserver started at http://127.10.225.131:38313/ using document root <none> and password file <none>
I20260806 23:52:46.363775 25495 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:46.363842 25495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:46.364048 25495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:46.366163 25495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "6c13441dc8a849b181a68c31e5ec106f"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "7739dac2cbd7265f05d6dc5823ae5faa"
server_key_iv: "7fa1b4533a159ae17afdb6b1aecb5f63"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.366700 25495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "6c13441dc8a849b181a68c31e5ec106f"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "7739dac2cbd7265f05d6dc5823ae5faa"
server_key_iv: "7fa1b4533a159ae17afdb6b1aecb5f63"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.371001 25495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:46.373730 25514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.375056 25495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 23:52:46.375164 25495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "6c13441dc8a849b181a68c31e5ec106f"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "7739dac2cbd7265f05d6dc5823ae5faa"
server_key_iv: "7fa1b4533a159ae17afdb6b1aecb5f63"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.375280 25495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:46.387498 25495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:46.388244 25495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:46.388428 25495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:46.389060 25495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:46.390075 25495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:46.390129 25495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.390169 25495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:46.390184 25495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.399399 25495 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:33683
I20260806 23:52:46.399456 25663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:33683 every 8 connection(s)
I20260806 23:52:46.400424 25495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:46.406430 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 25495
I20260806 23:52:46.406538 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:46.406795 11142 external_mini_cluster.cc:1428] Setting key 5d13f0e8e1fd0c752ffcf67209847580
I20260806 23:52:46.410769 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:46.402423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:41267 (local address 127.10.225.190:37001)
0806 23:52:46.402620 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:46.402624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:46.403923 (+ 1299us) server_negotiation.cc:407] Connection header received
0806 23:52:46.405802 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:46.405805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:46.405867 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:46.405956 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:46.406999 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.407643 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.408928 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.409182 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.409264 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:46.410311 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:46.410314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:46.410316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:46.410328 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:46.410348 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:46.410355 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:46.410416 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:46.410601 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 23:52:46.410652 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 23:52:46.411722 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:46.402729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:37001 (local address 127.10.225.131:41267)
0806 23:52:46.403643 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:46.403775 (+ 132us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:46.405514 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:46.406165 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:46.406173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:46.406270 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:46.406830 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:46.406847 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.407795 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.407798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:46.408809 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:46.408822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:46.409333 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:46.409337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:46.409510 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:46.410067 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:46.410068 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:46.410109 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:46.410178 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:46.410459 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:46.410469 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:46.410475 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:52:46.410599 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 23:52:46.410857 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
I20260806 23:52:46.412863 25665 heartbeater.cc:347] Connected to a master server at 127.10.225.190:37001
I20260806 23:52:46.413113 25665 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:46.413655 25665 heartbeater.cc:511] Master 127.10.225.190:37001 requested a full tablet report, sending...
I20260806 23:52:46.414705 25067 ts_manager.cc:194] Registered new tserver with Master: 6c13441dc8a849b181a68c31e5ec106f (127.10.225.131:33683)
I20260806 23:52:46.415259 25067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.131:41267
I20260806 23:52:46.420810 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 23:52:46.423575 25122 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:46.422760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.10.225.190:37001)
0806 23:52:46.422874 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:46.422878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:46.422964 (+ 86us) server_negotiation.cc:407] Connection header received
0806 23:52:46.423065 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:46.423067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:46.423118 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:46.423180 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:46.423462 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41506: BlockingRecv error: recv got EOF from 127.0.0.1:41506 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:52:46.423808 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 25148
I20260806 23:52:46.439808 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 25321
I20260806 23:52:46.458626 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 25495
I20260806 23:52:46.474923 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 25020
2026-08-06T23:52:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (965 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T23:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:46Z Disabled control of system clock
I20260806 23:52:46.510744 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:43161
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:43161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 23:52:46.633174 25677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:46.633472 25677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:46.633512 25677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:46.639453 25677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:46.639580 25677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:46.639596 25677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:46.639616 25677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:46.639635 25677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:46.645995 25677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:43161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:43161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:46.647214 25677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:46.648831 25677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:46.658034 25683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:46.658044 25687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:46.658044 25684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:46.658275 25677 server_base.cc:1034] running on GCE node
I20260806 23:52:46.658980 25677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:46.660214 25677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:46.661396 25677 hybrid_clock.cc:648] HybridClock initialized: now 1786060366661370 us; error 30 us; skew 500 ppm
I20260806 23:52:46.663456 25677 webserver.cc:460] Webserver started at http://127.10.225.190:46325/ using document root <none> and password file <none>
I20260806 23:52:46.664088 25677 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:46.664155 25677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:46.664350 25677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:46.666484 25677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "ea393a6cbedf46e5b516a29433172e47"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "b58474e91dd8553c4a3dc4374dbf9940"
server_key_iv: "bbba31c370fa92e0cd3f1ce14116df58"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.667033 25677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "ea393a6cbedf46e5b516a29433172e47"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "b58474e91dd8553c4a3dc4374dbf9940"
server_key_iv: "bbba31c370fa92e0cd3f1ce14116df58"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.670838 25677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:46.673496 25697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.675015 25677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:46.675170 25677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "ea393a6cbedf46e5b516a29433172e47"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "b58474e91dd8553c4a3dc4374dbf9940"
server_key_iv: "bbba31c370fa92e0cd3f1ce14116df58"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.675302 25677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:46.680485 25677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:46.681119 25677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:46.681279 25677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:46.688292 25677 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:43161
I20260806 23:52:46.688309 25765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:43161 every 8 connection(s)
I20260806 23:52:46.689304 25677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:46.692849 25766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:46.697659 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 25677
I20260806 23:52:46.697772 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:46.698037 11142 external_mini_cluster.cc:1428] Setting key 9fae5ec337f27f166017ee1d6795b36a
I20260806 23:52:46.699106 25766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47: Bootstrap starting.
I20260806 23:52:46.701676 25766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:46.702565 25766 log.cc:824] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:46.704034 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:46.699628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.10.225.190:43161)
0806 23:52:46.700419 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:46.700437 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:46.700474 (+ 37us) server_negotiation.cc:407] Connection header received
0806 23:52:46.701138 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:46.701154 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:46.701638 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:46.701965 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:46.702727 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:46.702734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:46.702749 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:46.702760 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:46.702818 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:46.702835 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:46.702934 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:46.703039 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 23:52:46.703275 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":147,"threads_started":1}
I20260806 23:52:46.704799 25766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47: No bootstrap required, opened a new log
I20260806 23:52:46.708659 25766 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea393a6cbedf46e5b516a29433172e47" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43161 } }
I20260806 23:52:46.708930 25766 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:46.709005 25766 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea393a6cbedf46e5b516a29433172e47, State: Initialized, Role: FOLLOWER
I20260806 23:52:46.709429 25766 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea393a6cbedf46e5b516a29433172e47" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43161 } }
I20260806 23:52:46.709568 25766 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:46.709628 25766 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:46.709719 25766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:46.710668 25766 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea393a6cbedf46e5b516a29433172e47" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43161 } }
I20260806 23:52:46.711002 25766 leader_election.cc:302] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea393a6cbedf46e5b516a29433172e47; no voters:
I20260806 23:52:46.711310 25766 leader_election.cc:288] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:46.711660 25772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:46.711944 25772 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [term 1 LEADER]: Becoming Leader. State: Replica: ea393a6cbedf46e5b516a29433172e47, State: Running, Role: LEADER
I20260806 23:52:46.712237 25772 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea393a6cbedf46e5b516a29433172e47" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43161 } }
I20260806 23:52:46.712679 25766 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:46.713837 25775 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea393a6cbedf46e5b516a29433172e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea393a6cbedf46e5b516a29433172e47" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43161 } } }
I20260806 23:52:46.713836 25776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea393a6cbedf46e5b516a29433172e47. Latest consensus state: current_term: 1 leader_uuid: "ea393a6cbedf46e5b516a29433172e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea393a6cbedf46e5b516a29433172e47" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43161 } } }
I20260806 23:52:46.713994 25775 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:46.714016 25776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:46.714396 25784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:46.717620 25784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:46.723807 25784 catalog_manager.cc:1347] Generated new cluster ID: d473c290b3ca435c94ed423387c7680f
I20260806 23:52:46.723927 25784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:46.743013 25784 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:46.743369 25784 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:46.750523 25784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea393a6cbedf46e5b516a29433172e47: Generated new TSK 0
I20260806 23:52:46.751245 25784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:52:46.758740 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43161
--builtin_ntp_servers=127.10.225.148:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 23:52:46.888186 25797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:46.888463 25797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:46.888523 25797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:46.892638 25797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:46.892704 25797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:46.892787 25797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:46.897933 25797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:46.899076 25797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:46.900517 25797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:46.910149 25808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:46.910164 25811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:46.910168 25809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:46.910477 25797 server_base.cc:1034] running on GCE node
I20260806 23:52:46.910882 25797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:46.911509 25797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:46.912696 25797 hybrid_clock.cc:648] HybridClock initialized: now 1786060366912678 us; error 34 us; skew 500 ppm
I20260806 23:52:46.914626 25797 webserver.cc:460] Webserver started at http://127.10.225.129:33963/ using document root <none> and password file <none>
I20260806 23:52:46.915237 25797 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:46.915305 25797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:46.915509 25797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:46.917693 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "99e41209621247fa9dd370ec85068eed"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "ea7d5275800fa2aad8f79ca6b3b61b9f"
server_key_iv: "03a9e68dabdd638e9f534266d6736e2c"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.918259 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "99e41209621247fa9dd370ec85068eed"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "ea7d5275800fa2aad8f79ca6b3b61b9f"
server_key_iv: "03a9e68dabdd638e9f534266d6736e2c"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.922117 25797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260806 23:52:46.924978 25821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.926414 25797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:52:46.926545 25797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "99e41209621247fa9dd370ec85068eed"
format_stamp: "Formatted at 2026-08-06 23:52:46 on dist-test-slave-nj21"
server_key: "ea7d5275800fa2aad8f79ca6b3b61b9f"
server_key_iv: "03a9e68dabdd638e9f534266d6736e2c"
server_key_version: "encryptionkey@0"
I20260806 23:52:46.926651 25797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:46.933194 25797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:46.934136 25797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:46.934324 25797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:46.935214 25797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:46.936424 25797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:46.936479 25797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.936527 25797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:46.936543 25797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:46.947043 25797 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:36509
I20260806 23:52:46.947068 25973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:36509 every 8 connection(s)
I20260806 23:52:46.948441 25797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:46.955211 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:46.950632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:34703 (local address 127.10.225.190:43161)
0806 23:52:46.950766 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:46.950770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:46.951957 (+ 1187us) server_negotiation.cc:407] Connection header received
0806 23:52:46.953299 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:46.953302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:46.953346 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:46.953426 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:46.954699 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:46.954703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:46.954705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:46.954718 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:46.954739 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:46.954747 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:46.954832 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:46.955045 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 23:52:46.955093 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:52:46.956148 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:46.950980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:43161 (local address 127.10.225.129:34703)
0806 23:52:46.951808 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:46.951853 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:46.953061 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:46.953629 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:46.953636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:46.953724 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:46.954433 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:46.954435 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:46.954478 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:46.954547 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:46.954861 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:46.954871 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:46.954876 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 23:52:46.955036 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 23:52:46.955238 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":149,"threads_started":1}
I20260806 23:52:46.956949 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 25797
I20260806 23:52:46.957048 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:46.957366 11142 external_mini_cluster.cc:1428] Setting key c057785faa258880f2ddb68c999c31b5
I20260806 23:52:46.957535 25975 heartbeater.cc:347] Connected to a master server at 127.10.225.190:43161
I20260806 23:52:46.957780 25975 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:46.958225 25975 heartbeater.cc:511] Master 127.10.225.190:43161 requested a full tablet report, sending...
I20260806 23:52:46.959239 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43161
--builtin_ntp_servers=127.10.225.148:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 23:52:46.959937 25718 ts_manager.cc:194] Registered new tserver with Master: 99e41209621247fa9dd370ec85068eed (127.10.225.129:36509)
W20260806 23:52:47.084731 25979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:47.085023 25979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:47.085065 25979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:47.089552 25979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:47.089643 25979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:47.089706 25979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:47.095881 25979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:47.097107 25979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:47.098590 25979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:47.108860 25988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:47.108927 25992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:47.109139 25987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:47.109453 25979 server_base.cc:1034] running on GCE node
I20260806 23:52:47.109905 25979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:47.110606 25979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:47.111838 25979 hybrid_clock.cc:648] HybridClock initialized: now 1786060367111784 us; error 47 us; skew 500 ppm
I20260806 23:52:47.114573 25979 webserver.cc:460] Webserver started at http://127.10.225.130:41935/ using document root <none> and password file <none>
I20260806 23:52:47.115227 25979 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:47.115300 25979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:47.115523 25979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:47.117764 25979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "fb0f64c18eb84d41b0d2fafd54964d81"
format_stamp: "Formatted at 2026-08-06 23:52:47 on dist-test-slave-nj21"
server_key: "85610f02731e3de2062a4ac6c0458aa0"
server_key_iv: "bc8acfaf2e12487844106f7ee18beba7"
server_key_version: "encryptionkey@0"
I20260806 23:52:47.118345 25979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "fb0f64c18eb84d41b0d2fafd54964d81"
format_stamp: "Formatted at 2026-08-06 23:52:47 on dist-test-slave-nj21"
server_key: "85610f02731e3de2062a4ac6c0458aa0"
server_key_iv: "bc8acfaf2e12487844106f7ee18beba7"
server_key_version: "encryptionkey@0"
I20260806 23:52:47.122531 25979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:52:47.125896 26001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:47.127460 25979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:47.127612 25979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "fb0f64c18eb84d41b0d2fafd54964d81"
format_stamp: "Formatted at 2026-08-06 23:52:47 on dist-test-slave-nj21"
server_key: "85610f02731e3de2062a4ac6c0458aa0"
server_key_iv: "bc8acfaf2e12487844106f7ee18beba7"
server_key_version: "encryptionkey@0"
I20260806 23:52:47.127799 25979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:47.133152 25979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:47.134236 25979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:47.134455 25979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:47.135088 25979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:47.136183 25979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:47.136234 25979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:47.136291 25979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:47.136322 25979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:47.146507 25979 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:35475
I20260806 23:52:47.146565 26171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:35475 every 8 connection(s)
I20260806 23:52:47.147584 25979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:47.147941 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 25979
I20260806 23:52:47.148084 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:47.148496 11142 external_mini_cluster.cc:1428] Setting key af4b2528593417c82c0060ecea6fa08a
I20260806 23:52:47.151134 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43161
--builtin_ntp_servers=127.10.225.148:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 23:52:47.156558 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:47.149631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:53091 (local address 127.10.225.190:43161)
0806 23:52:47.149781 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:47.149785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:47.151741 (+ 1956us) server_negotiation.cc:407] Connection header received
0806 23:52:47.153887 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:47.153893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:47.153963 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:47.154172 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:47.155852 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:47.155856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:47.155858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:47.155875 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:47.155903 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:47.155914 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:47.156021 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:47.156317 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 23:52:47.156383 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:52:47.157532 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:47.150180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:43161 (local address 127.10.225.130:53091)
0806 23:52:47.151451 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:47.151522 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:47.153425 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:47.154397 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:47.154412 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:47.154638 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:47.155437 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:47.155439 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:47.155496 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:47.155630 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:47.155995 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:47.156005 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:47.156012 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 23:52:47.156324 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 23:52:47.156634 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":253,"threads_started":1}
I20260806 23:52:47.159193 26173 heartbeater.cc:347] Connected to a master server at 127.10.225.190:43161
I20260806 23:52:47.159536 26173 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:47.160209 26173 heartbeater.cc:511] Master 127.10.225.190:43161 requested a full tablet report, sending...
I20260806 23:52:47.161988 25718 ts_manager.cc:194] Registered new tserver with Master: fb0f64c18eb84d41b0d2fafd54964d81 (127.10.225.130:35475)
W20260806 23:52:47.276764 26182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:47.277091 26182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:47.277148 26182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:47.282298 26182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:47.282369 26182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:47.282433 26182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:47.288911 26182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:43161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:47.290164 26182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:47.292452 26182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:47.303334 26188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:47.303335 26189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:47.303555 26192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:47.303467 26182 server_base.cc:1034] running on GCE node
I20260806 23:52:47.304064 26182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:47.304754 26182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:47.305931 26182 hybrid_clock.cc:648] HybridClock initialized: now 1786060367305913 us; error 28 us; skew 500 ppm
I20260806 23:52:47.307926 26182 webserver.cc:460] Webserver started at http://127.10.225.131:37243/ using document root <none> and password file <none>
I20260806 23:52:47.308522 26182 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:47.308593 26182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:47.308810 26182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:47.311008 26182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "2cd0601a394645e1b89ebe93648fc6ad"
format_stamp: "Formatted at 2026-08-06 23:52:47 on dist-test-slave-nj21"
server_key: "4a9371ddbc5f1a13994d78da0c7ef169"
server_key_iv: "a9b179e5b926ba0cd2bf23ee36296559"
server_key_version: "encryptionkey@0"
I20260806 23:52:47.311599 26182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "2cd0601a394645e1b89ebe93648fc6ad"
format_stamp: "Formatted at 2026-08-06 23:52:47 on dist-test-slave-nj21"
server_key: "4a9371ddbc5f1a13994d78da0c7ef169"
server_key_iv: "a9b179e5b926ba0cd2bf23ee36296559"
server_key_version: "encryptionkey@0"
I20260806 23:52:47.315969 26182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:52:47.318755 26204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:47.320391 26182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:47.320521 26182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "2cd0601a394645e1b89ebe93648fc6ad"
format_stamp: "Formatted at 2026-08-06 23:52:47 on dist-test-slave-nj21"
server_key: "4a9371ddbc5f1a13994d78da0c7ef169"
server_key_iv: "a9b179e5b926ba0cd2bf23ee36296559"
server_key_version: "encryptionkey@0"
I20260806 23:52:47.320639 26182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:47.325340 26182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:47.325991 26182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:47.326152 26182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:47.326694 26182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:47.327641 26182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:47.327749 26182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:47.327912 26182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:47.327962 26182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:47.337999 26182 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:45565
I20260806 23:52:47.338024 26359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:45565 every 8 connection(s)
I20260806 23:52:47.339435 26182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:47.341231 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 26182
I20260806 23:52:47.341341 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:47.341646 11142 external_mini_cluster.cc:1428] Setting key 60b95bf796753039b36752f02654db43
I20260806 23:52:47.346357 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:47.341520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:38153 (local address 127.10.225.190:43161)
0806 23:52:47.341687 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:47.341692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:47.342990 (+ 1298us) server_negotiation.cc:407] Connection header received
0806 23:52:47.344481 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:47.344484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:47.344525 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:47.344612 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:47.345806 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:47.345811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:47.345813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:52:47.345836 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:47.345870 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:52:47.345881 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:47.346015 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:47.346195 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 23:52:47.346241 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 23:52:47.347223 26364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:47.341972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:43161 (local address 127.10.225.131:38153)
0806 23:52:47.342856 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:47.342901 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:47.344264 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:47.344813 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:47.344829 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:47.344920 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:47.345445 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:52:47.345447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:47.345488 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:52:47.345556 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:47.346005 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:47.346013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:47.346018 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 23:52:47.346167 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 23:52:47.346375 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":134,"threads_started":1}
I20260806 23:52:47.348354 26360 heartbeater.cc:347] Connected to a master server at 127.10.225.190:43161
I20260806 23:52:47.348578 26360 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:47.349009 26360 heartbeater.cc:511] Master 127.10.225.190:43161 requested a full tablet report, sending...
I20260806 23:52:47.350049 25718 ts_manager.cc:194] Registered new tserver with Master: 2cd0601a394645e1b89ebe93648fc6ad (127.10.225.131:45565)
I20260806 23:52:47.356256 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 23:52:47.359910 25769 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:52:47.358832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.10.225.190:43161)
0806 23:52:47.358997 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:47.359001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:47.359252 (+ 251us) server_negotiation.cc:407] Connection header received
0806 23:52:47.359434 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:47.359437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:47.359477 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:47.359549 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:47.359779 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52778: BlockingRecv error: recv got EOF from 127.0.0.1:52778 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 23:52:47.360463 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 25797
I20260806 23:52:47.375098 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 25979
I20260806 23:52:47.398655 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 26182
I20260806 23:52:47.410252 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 25677
2026-08-06T23:52:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (939 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:52:47 dist-test-slave-nj21 krb5kdc[26377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:52:47 dist-test-slave-nj21 krb5kdc[26377](info): set up 2 sockets
Aug 06 23:52:47 dist-test-slave-nj21 krb5kdc[26377](info): commencing operation
krb5kdc: starting...
W20260806 23:52:48.575554 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:48 dist-test-slave-nj21 krb5kdc[26377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:52:48.728803 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34187
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:34187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:48.852077 26395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:48.852370 26395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:48.852429 26395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:48.856514 26395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:52:48.856606 26395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:48.856634 26395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:48.856658 26395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:52:48.856688 26395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:52:48.862146 26395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:34187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:48.863308 26395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:48.864380 26395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:48.873348 26400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:48.873356 26403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:48.873394 26401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:48.874203 26395 server_base.cc:1034] running on GCE node
I20260806 23:52:48.874756 26395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:48.875941 26395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:48.877120 26395 hybrid_clock.cc:648] HybridClock initialized: now 1786060368877096 us; error 37 us; skew 500 ppm
Aug 06 23:52:48 dist-test-slave-nj21 krb5kdc[26377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060368, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:48.880005 26395 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:52:48.880954 26395 webserver.cc:460] Webserver started at http://127.10.225.190:35653/ using document root <none> and password file <none>
I20260806 23:52:48.881512 26395 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:48.881582 26395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:48.881788 26395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:48.883889 26395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "84c508fdcc1747649d73b7600f06ef89"
format_stamp: "Formatted at 2026-08-06 23:52:48 on dist-test-slave-nj21"
server_key: "ef107fdf8a553713aa045c883f5bb2c6"
server_key_iv: "53b1238ad53012314dca02f3e4656d5e"
server_key_version: "encryptionkey@0"
I20260806 23:52:48.884446 26395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "84c508fdcc1747649d73b7600f06ef89"
format_stamp: "Formatted at 2026-08-06 23:52:48 on dist-test-slave-nj21"
server_key: "ef107fdf8a553713aa045c883f5bb2c6"
server_key_iv: "53b1238ad53012314dca02f3e4656d5e"
server_key_version: "encryptionkey@0"
I20260806 23:52:48.888172 26395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:48.890645 26412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:48.892324 26395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:52:48.892458 26395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "84c508fdcc1747649d73b7600f06ef89"
format_stamp: "Formatted at 2026-08-06 23:52:48 on dist-test-slave-nj21"
server_key: "ef107fdf8a553713aa045c883f5bb2c6"
server_key_iv: "53b1238ad53012314dca02f3e4656d5e"
server_key_version: "encryptionkey@0"
I20260806 23:52:48.892560 26395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:48.910001 26395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:48.914227 26395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:48.914436 26395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:48.921695 26395 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:34187
I20260806 23:52:48.921977 26488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:34187 every 8 connection(s)
I20260806 23:52:48.922808 26395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:52:48.925740 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:48.928009 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 26395
I20260806 23:52:48.928125 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:52:48.928376 11142 external_mini_cluster.cc:1428] Setting key c53a55f5a07f1d39802e76a2157198ec
I20260806 23:52:48.932734 26489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89: Bootstrap starting.
I20260806 23:52:48.935570 26489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:48.936395 26489 log.cc:824] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:52:48 dist-test-slave-nj21 krb5kdc[26377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:48.938803 26489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89: No bootstrap required, opened a new log
I20260806 23:52:48.942907 26489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c508fdcc1747649d73b7600f06ef89" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34187 } }
I20260806 23:52:48.943197 26489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:48.943264 26489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c508fdcc1747649d73b7600f06ef89, State: Initialized, Role: FOLLOWER
I20260806 23:52:48.943781 26489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c508fdcc1747649d73b7600f06ef89" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34187 } }
I20260806 23:52:48.944023 26489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:52:48.944104 26489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:52:48.944149 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:48.929847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.10.225.190:34187)
0806 23:52:48.930303 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:48.930324 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:48.930367 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:52:48.931173 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:48.931198 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:48.931629 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:48.932047 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:48.933150 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:48.934638 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:48.935873 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:48.936364 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:48.936500 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:48.939427 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:48.939432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:48.939447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:48.939474 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:48.941142 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:48.941649 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:48.941654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:48.941659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:48.941772 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:48.942032 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:48.942035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:48.942036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:48.942444 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:48.942596 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:48.942904 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 23:52:48.943193 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260806 23:52:48.944208 26489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:48.945438 26489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c508fdcc1747649d73b7600f06ef89" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34187 } }
I20260806 23:52:48.945791 26489 leader_election.cc:302] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c508fdcc1747649d73b7600f06ef89; no voters:
I20260806 23:52:48.946130 26489 leader_election.cc:288] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:52:48.946228 26495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:48.946488 26495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [term 1 LEADER]: Becoming Leader. State: Replica: 84c508fdcc1747649d73b7600f06ef89, State: Running, Role: LEADER
I20260806 23:52:48.946807 26495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c508fdcc1747649d73b7600f06ef89" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34187 } }
I20260806 23:52:48.947575 26489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:52:48.948623 26497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84c508fdcc1747649d73b7600f06ef89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c508fdcc1747649d73b7600f06ef89" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34187 } } }
I20260806 23:52:48.948753 26497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:48.949265 26498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c508fdcc1747649d73b7600f06ef89. Latest consensus state: current_term: 1 leader_uuid: "84c508fdcc1747649d73b7600f06ef89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c508fdcc1747649d73b7600f06ef89" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34187 } } }
I20260806 23:52:48.949357 26498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89 [sys.catalog]: This master's current role is: LEADER
I20260806 23:52:48.949770 26508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:52:48.953073 26508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:52:48.958920 26508 catalog_manager.cc:1347] Generated new cluster ID: ba95bf19136548e282967c97112f8081
I20260806 23:52:48.959009 26508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:52:48.966089 26508 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:52:48.967002 26508 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:52:48.978478 26508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89: Generated new TSK 0
I20260806 23:52:48.979380 26508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:52:49.022209 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34187
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:49.156680 26524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:49.156980 26524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:49.157042 26524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:49.161158 26524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:49.161226 26524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:49.161306 26524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:52:49.166709 26524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:49.167882 26524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:49.168972 26524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:49.179015 26530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:49.179033 26533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:49.179100 26524 server_base.cc:1034] running on GCE node
W20260806 23:52:49.179032 26531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:49.179801 26524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:49.180495 26524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:49.181674 26524 hybrid_clock.cc:648] HybridClock initialized: now 1786060369181657 us; error 30 us; skew 500 ppm
Aug 06 23:52:49 dist-test-slave-nj21 krb5kdc[26377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060369, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:49.197890 26524 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:52:49.198999 26524 webserver.cc:460] Webserver started at http://127.10.225.129:36977/ using document root <none> and password file <none>
I20260806 23:52:49.199605 26524 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:49.199735 26524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:49.199980 26524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:49.202365 26524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "5826edd2c6a24e9e8b08ea4b73e68e11"
format_stamp: "Formatted at 2026-08-06 23:52:49 on dist-test-slave-nj21"
server_key: "ffa2cd5b778012844004bf07fa5d2524"
server_key_iv: "933489cf09caf10623eaabf90e6a4672"
server_key_version: "encryptionkey@0"
I20260806 23:52:49.203011 26524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "5826edd2c6a24e9e8b08ea4b73e68e11"
format_stamp: "Formatted at 2026-08-06 23:52:49 on dist-test-slave-nj21"
server_key: "ffa2cd5b778012844004bf07fa5d2524"
server_key_iv: "933489cf09caf10623eaabf90e6a4672"
server_key_version: "encryptionkey@0"
I20260806 23:52:49.207515 26524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:52:49.210287 26543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:49.211585 26524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:52:49.211731 26524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "5826edd2c6a24e9e8b08ea4b73e68e11"
format_stamp: "Formatted at 2026-08-06 23:52:49 on dist-test-slave-nj21"
server_key: "ffa2cd5b778012844004bf07fa5d2524"
server_key_iv: "933489cf09caf10623eaabf90e6a4672"
server_key_version: "encryptionkey@0"
I20260806 23:52:49.211838 26524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:49.231983 26524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:49.235471 26524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:49.235651 26524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:49.236423 26524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:49.237926 26524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:49.238003 26524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:49.238046 26524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:49.238061 26524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:49.247359 26524 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:42855
I20260806 23:52:49.247478 26692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:42855 every 8 connection(s)
I20260806 23:52:49.248472 26524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:52:49.251379 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 26524
I20260806 23:52:49.251504 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:52:49.251848 11142 external_mini_cluster.cc:1428] Setting key d588e7715daa38ae6a2e952dd0770f0e
Aug 06 23:52:49 dist-test-slave-nj21 krb5kdc[26377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060369, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:52:49.266047 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.250519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:53979 (local address 127.10.225.190:34187)
0806 23:52:49.250661 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:49.250665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:49.251532 (+ 867us) server_negotiation.cc:407] Connection header received
0806 23:52:49.253235 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:49.253239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:49.253431 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:49.253689 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:49.255450 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.256209 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.258205 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.258651 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.258805 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.262298 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:49.262301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:49.262303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:49.262331 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:49.263812 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:49.264409 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:49.264412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:49.264413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:49.264499 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:49.264849 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:49.264851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:49.264853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:49.265066 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:49.265174 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:49.265758 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 23:52:49.265906 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:52:49.266815 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.250791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34187 (local address 127.10.225.129:53979)
0806 23:52:49.251353 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:49.251417 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:49.253025 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:49.253978 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:49.253990 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:49.254627 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:49.255181 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.255198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.256417 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.256422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.258007 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.258021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.258895 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.258901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.259218 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.260162 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:49.260182 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:49.262054 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:49.263937 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:49.263945 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:49.263963 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:49.264315 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:49.264589 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:49.264592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:49.264593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:49.264738 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:49.265185 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:49.265192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:49.265474 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 23:52:49.265644 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 23:52:49.265940 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260806 23:52:49.268136 26693 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34187
I20260806 23:52:49.268392 26693 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:49.268982 26693 heartbeater.cc:511] Master 127.10.225.190:34187 requested a full tablet report, sending...
I20260806 23:52:49.270728 26438 ts_manager.cc:194] Registered new tserver with Master: 5826edd2c6a24e9e8b08ea4b73e68e11 (127.10.225.129:42855)
I20260806 23:52:49.272240 26438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:53979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:52:49.283084 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34187
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:49.401274 26700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:49.401566 26700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:49.401611 26700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:49.406375 26700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:49.406469 26700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:49.406536 26700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:52:49.413599 26700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:49.414783 26700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:49.415882 26700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:52:49.426120 26700 server_base.cc:1034] running on GCE node
W20260806 23:52:49.426067 26710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:49.426079 26713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:49.426071 26709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:49.426719 26700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:49.427382 26700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:49.428565 26700 hybrid_clock.cc:648] HybridClock initialized: now 1786060369428525 us; error 52 us; skew 500 ppm
Aug 06 23:52:49 dist-test-slave-nj21 krb5kdc[26377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060369, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:49.431450 26700 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:52:49.432395 26700 webserver.cc:460] Webserver started at http://127.10.225.130:46767/ using document root <none> and password file <none>
I20260806 23:52:49.432973 26700 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:49.433036 26700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:49.433233 26700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:49.435367 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "5fef0017ee8846db96520a7765f528f8"
format_stamp: "Formatted at 2026-08-06 23:52:49 on dist-test-slave-nj21"
server_key: "74331ab51dc8fb3e942bbe7c56102708"
server_key_iv: "77544b3363346e1a332625604bfade65"
server_key_version: "encryptionkey@0"
I20260806 23:52:49.435974 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "5fef0017ee8846db96520a7765f528f8"
format_stamp: "Formatted at 2026-08-06 23:52:49 on dist-test-slave-nj21"
server_key: "74331ab51dc8fb3e942bbe7c56102708"
server_key_iv: "77544b3363346e1a332625604bfade65"
server_key_version: "encryptionkey@0"
I20260806 23:52:49.440579 26700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:52:49.443908 26724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:49.445520 26700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:52:49.445649 26700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "5fef0017ee8846db96520a7765f528f8"
format_stamp: "Formatted at 2026-08-06 23:52:49 on dist-test-slave-nj21"
server_key: "74331ab51dc8fb3e942bbe7c56102708"
server_key_iv: "77544b3363346e1a332625604bfade65"
server_key_version: "encryptionkey@0"
I20260806 23:52:49.445768 26700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:49.469403 26700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:49.473484 26700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:49.473680 26700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:49.474398 26700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:49.475462 26700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:49.475519 26700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:49.475575 26700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:49.475602 26700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:49.484905 26700 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:42623
I20260806 23:52:49.484977 26871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:42623 every 8 connection(s)
I20260806 23:52:49.485831 26700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:52:49.491449 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 26700
I20260806 23:52:49.491552 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:52:49.491874 11142 external_mini_cluster.cc:1428] Setting key 5e19309f37e2d114be0194567c3a0d22
Aug 06 23:52:49 dist-test-slave-nj21 krb5kdc[26377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060369, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:49.504633 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.487751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:33591 (local address 127.10.225.190:34187)
0806 23:52:49.487892 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:49.487896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:49.488751 (+ 855us) server_negotiation.cc:407] Connection header received
0806 23:52:49.490549 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:49.490552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:49.490608 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:49.490680 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:49.492701 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.493539 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.495999 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.496299 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.496405 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.500352 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:49.500359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:49.500362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:49.500402 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:49.502016 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:49.502750 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:49.502753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:49.502754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:49.502838 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:49.503289 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:49.503292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:49.503293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:49.503528 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:49.503614 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:49.504357 (+ 743us) server_negotiation.cc:299] Negotiation successful
0806 23:52:49.504499 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:52:49.505748 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.488118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34187 (local address 127.10.225.130:33591)
0806 23:52:49.488600 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:49.488645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:49.490182 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:49.490989 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:49.490999 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:49.491524 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:49.492479 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.492506 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.493723 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.493728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.495777 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.495796 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.496497 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.496593 (+ 96us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.496964 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.498051 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:49.498070 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:49.500052 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:49.502161 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:49.502168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:49.502189 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:49.502615 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:49.502926 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:49.502930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:49.502938 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:49.503131 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:49.503642 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:49.503650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:49.504048 (+ 398us) client_negotiation.cc:769] Sending connection context
0806 23:52:49.504276 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 23:52:49.504691 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260806 23:52:49.507004 26872 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34187
I20260806 23:52:49.507261 26872 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
I20260806 23:52:49.507815 26872 heartbeater.cc:511] Master 127.10.225.190:34187 requested a full tablet report, sending...
I20260806 23:52:49.508934 26438 ts_manager.cc:194] Registered new tserver with Master: 5fef0017ee8846db96520a7765f528f8 (127.10.225.130:42623)
I20260806 23:52:49.509536 26438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:33591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:52:49.526292 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34187
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:52:49.653414 26879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:52:49.653757 26879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:52:49.653810 26879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:52:49.658546 26879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:52:49.658610 26879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:52:49.658670 26879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:52:49.664242 26879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:52:49.665328 26879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:52:49.666333 26879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:52:49.675872 26886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:52:49.675940 26885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:49.676102 26879 server_base.cc:1034] running on GCE node
W20260806 23:52:49.676331 26888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:52:49.676795 26879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:52:49.677639 26879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:52:49.678862 26879 hybrid_clock.cc:648] HybridClock initialized: now 1786060369678836 us; error 42 us; skew 500 ppm
Aug 06 23:52:49 dist-test-slave-nj21 krb5kdc[26377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060369, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:52:49.682267 26879 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:52:49.683162 26879 webserver.cc:460] Webserver started at http://127.10.225.131:36889/ using document root <none> and password file <none>
I20260806 23:52:49.683774 26879 fs_manager.cc:359] Metadata directory not provided
I20260806 23:52:49.683851 26879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:52:49.684060 26879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:52:49.686118 26879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "cfe06faac6af44c28647b8aa8444f0f2"
format_stamp: "Formatted at 2026-08-06 23:52:49 on dist-test-slave-nj21"
server_key: "e1fd13788d2d9eb5c00dd0a00fcdc156"
server_key_iv: "30280ff4d64fddd5ef77f31d97fa47d8"
server_key_version: "encryptionkey@0"
I20260806 23:52:49.686693 26879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "cfe06faac6af44c28647b8aa8444f0f2"
format_stamp: "Formatted at 2026-08-06 23:52:49 on dist-test-slave-nj21"
server_key: "e1fd13788d2d9eb5c00dd0a00fcdc156"
server_key_iv: "30280ff4d64fddd5ef77f31d97fa47d8"
server_key_version: "encryptionkey@0"
I20260806 23:52:49.690608 26879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 23:52:49.693542 26897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:49.694739 26879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:52:49.694856 26879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "cfe06faac6af44c28647b8aa8444f0f2"
format_stamp: "Formatted at 2026-08-06 23:52:49 on dist-test-slave-nj21"
server_key: "e1fd13788d2d9eb5c00dd0a00fcdc156"
server_key_iv: "30280ff4d64fddd5ef77f31d97fa47d8"
server_key_version: "encryptionkey@0"
I20260806 23:52:49.694968 26879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:52:49.711850 26879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:52:49.716657 26879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:52:49.716923 26879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:52:49.717816 26879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:52:49.718825 26879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:52:49.718883 26879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:49.718938 26879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:52:49.718966 26879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:52:49.728302 26879 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:37813
I20260806 23:52:49.728330 27044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:37813 every 8 connection(s)
I20260806 23:52:49.729236 26879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:52:49.735152 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 26879
I20260806 23:52:49.735296 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:52:49.735575 11142 external_mini_cluster.cc:1428] Setting key cbd73952a707b49fea27fa8a25e7eb7c
Aug 06 23:52:49 dist-test-slave-nj21 krb5kdc[26377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060369, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:52:49.744060 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.731155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:51059 (local address 127.10.225.190:34187)
0806 23:52:49.731307 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:49.731310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:49.732184 (+ 874us) server_negotiation.cc:407] Connection header received
0806 23:52:49.733556 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:49.733559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:49.733610 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:49.733692 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:49.735299 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.735922 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.737408 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.737837 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.737926 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.740537 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:49.740540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:49.740543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:49.740569 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:49.741805 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:49.742546 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:49.742549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:49.742550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:49.742630 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:49.742925 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:49.742927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:49.742928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:49.743112 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:49.743189 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:49.743806 (+ 617us) server_negotiation.cc:299] Negotiation successful
0806 23:52:49.743932 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:52:49.744904 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.731485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34187 (local address 127.10.225.131:51059)
0806 23:52:49.732029 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:49.732073 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:49.733334 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:49.733887 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:49.733900 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:49.734453 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:52:49.735108 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.735130 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.736063 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.736067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.737202 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.737214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.737985 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.737988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.738140 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.738732 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:52:49.738750 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:52:49.740352 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:52:49.741918 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:49.741924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:49.741941 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:49.742469 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:49.742703 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:52:49.742705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:52:49.742706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:52:49.742845 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:52:49.743232 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:52:49.743238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:52:49.743534 (+ 296us) client_negotiation.cc:769] Sending connection context
0806 23:52:49.743740 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 23:52:49.744057 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":170,"threads_started":1}
I20260806 23:52:49.746026 27045 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34187
I20260806 23:52:49.746263 27045 heartbeater.cc:464] Registering TS with master...
I20260806 23:52:49.746769 27045 heartbeater.cc:511] Master 127.10.225.190:34187 requested a full tablet report, sending...
I20260806 23:52:49.747862 26438 ts_manager.cc:194] Registered new tserver with Master: cfe06faac6af44c28647b8aa8444f0f2 (127.10.225.131:37813)
I20260806 23:52:49.748447 26438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:51059
I20260806 23:52:49.749481 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:52:49.758613 26492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.751701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.10.225.190:34187)
0806 23:52:49.751802 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:49.751806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:49.752001 (+ 195us) server_negotiation.cc:407] Connection header received
0806 23:52:49.752090 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:49.752092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:49.752128 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:49.752190 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:52:49.752785 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.753279 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.754143 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.754453 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.754538 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.755838 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:52:49.755841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:52:49.755843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:52:49.755872 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:52:49.757121 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:49.757589 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:49.757592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:49.757593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:49.757673 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:52:49.757885 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:52:49.757886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:52:49.757888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:52:49.758088 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:52:49.758188 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:49.758403 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 23:52:49.758494 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 23:52:49.761445 26438 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:52:49.764526 26438 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:52:49.779495 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.773738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:37813 (local address 127.0.0.1:51876)
0806 23:52:49.774107 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:49.774122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:49.774248 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:49.775331 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:49.775335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:49.775354 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:49.775443 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.775462 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.777076 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.777079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.777917 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.777926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.778956 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.778959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.779064 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.779101 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 23:52:49.779207 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 23:52:49.779332 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"spinlock_wait_cycles":384,"thread_start_us":90,"threads_started":1}
I20260806 23:52:49.779497 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.774396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.10.225.131:37813)
0806 23:52:49.774844 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:49.774855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:49.774889 (+ 34us) server_negotiation.cc:407] Connection header received
0806 23:52:49.774986 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:49.774991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:49.775182 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:49.775602 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:49.775779 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.776949 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.778038 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.778857 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.778967 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.779105 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:49.779244 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 23:52:49.779344 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":162,"threads_started":1}
I20260806 23:52:49.780022 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.773601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:42855 (local address 127.0.0.1:54948)
0806 23:52:49.774059 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:49.774108 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:49.774249 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:49.775040 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:49.775051 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:49.775077 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:49.775235 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.775243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.776962 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.776965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.778421 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.778434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.779510 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.779515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.779662 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.779713 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 23:52:49.779775 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 23:52:49.779879 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":151,"spinlock_wait_cycles":3456,"thread_start_us":84,"threads_started":1}
I20260806 23:52:49.780032 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.774302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:42623 (local address 127.0.0.1:54260)
0806 23:52:49.774524 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:49.774537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:49.774618 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:49.775503 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:49.775506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:49.775522 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:49.775629 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.775636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.776864 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.776868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.778421 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.778434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.779511 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.779515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.779692 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.779715 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:52:49.779775 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 23:52:49.779907 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":184,"thread_start_us":109,"threads_started":1}
I20260806 23:52:49.780531 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.774749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54260 (local address 127.10.225.130:42623)
0806 23:52:49.775063 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:49.775077 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:49.775098 (+ 21us) server_negotiation.cc:407] Connection header received
0806 23:52:49.775175 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:49.775180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:49.775361 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:49.775826 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:49.775983 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.776740 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.778565 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.779370 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.779899 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.780027 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:49.780078 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 23:52:49.780172 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":118,"threads_started":1}
I20260806 23:52:49.780933 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.773757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.10.225.129:42855)
0806 23:52:49.774528 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:49.774536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:49.774562 (+ 26us) server_negotiation.cc:407] Connection header received
0806 23:52:49.774644 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:49.774649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:49.774854 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:49.775405 (+ 551us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:49.775532 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.776853 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.778565 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.779370 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.780424 (+ 1054us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.780549 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:49.780684 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 23:52:49.780796 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":64,"threads_started":1}
I20260806 23:52:49.783152 26954 tablet_service.cc:1467] Processing CreateTablet for tablet f43d50f077934651b373f291cadaf76a (DEFAULT_TABLE table=test-table [id=3d5f05b76d494443b8a7a9c7392f31b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:49.783151 26783 tablet_service.cc:1467] Processing CreateTablet for tablet f43d50f077934651b373f291cadaf76a (DEFAULT_TABLE table=test-table [id=3d5f05b76d494443b8a7a9c7392f31b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:49.784374 26783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43d50f077934651b373f291cadaf76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:49.784379 26954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43d50f077934651b373f291cadaf76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:49.786634 26604 tablet_service.cc:1467] Processing CreateTablet for tablet f43d50f077934651b373f291cadaf76a (DEFAULT_TABLE table=test-table [id=3d5f05b76d494443b8a7a9c7392f31b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:52:49.787525 26604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f43d50f077934651b373f291cadaf76a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:52:49.793136 27068 tablet_bootstrap.cc:492] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: Bootstrap starting.
I20260806 23:52:49.794185 27069 tablet_bootstrap.cc:492] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: Bootstrap starting.
I20260806 23:52:49.795472 27068 tablet_bootstrap.cc:654] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:49.796021 27072 tablet_bootstrap.cc:492] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11: Bootstrap starting.
I20260806 23:52:49.796526 27069 tablet_bootstrap.cc:654] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:49.796892 27068 log.cc:824] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:49.797417 27069 log.cc:824] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:49.798346 27072 tablet_bootstrap.cc:654] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11: Neither blocks nor log segments found. Creating new log.
I20260806 23:52:49.799132 27072 log.cc:824] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11: Log is configured to *not* fsync() on all Append() calls
I20260806 23:52:49.799301 27069 tablet_bootstrap.cc:492] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: No bootstrap required, opened a new log
I20260806 23:52:49.799449 27068 tablet_bootstrap.cc:492] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: No bootstrap required, opened a new log
I20260806 23:52:49.799503 27069 ts_tablet_manager.cc:1397] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:52:49.799650 27068 ts_tablet_manager.cc:1397] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 23:52:49.801381 27072 tablet_bootstrap.cc:492] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11: No bootstrap required, opened a new log
I20260806 23:52:49.801645 27072 ts_tablet_manager.cc:1397] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:52:49.802630 27069 raft_consensus.cc:348] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } }
I20260806 23:52:49.802915 27069 raft_consensus.cc:374] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:49.802976 27069 raft_consensus.cc:729] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fef0017ee8846db96520a7765f528f8, State: Initialized, Role: FOLLOWER
I20260806 23:52:49.803450 27069 consensus_queue.cc:260] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } }
I20260806 23:52:49.803457 27068 raft_consensus.cc:348] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } }
I20260806 23:52:49.803753 27068 raft_consensus.cc:374] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:49.803838 27068 raft_consensus.cc:729] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe06faac6af44c28647b8aa8444f0f2, State: Initialized, Role: FOLLOWER
I20260806 23:52:49.804256 27068 consensus_queue.cc:260] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } }
I20260806 23:52:49.804436 27069 ts_tablet_manager.cc:1428] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260806 23:52:49.804801 26872 heartbeater.cc:503] Master 127.10.225.190:34187 was elected leader, sending a full tablet report...
I20260806 23:52:49.804903 27045 heartbeater.cc:503] Master 127.10.225.190:34187 was elected leader, sending a full tablet report...
I20260806 23:52:49.805222 27068 ts_tablet_manager.cc:1428] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260806 23:52:49.806272 27072 raft_consensus.cc:348] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } }
I20260806 23:52:49.806524 27072 raft_consensus.cc:374] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:52:49.806576 27072 raft_consensus.cc:729] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5826edd2c6a24e9e8b08ea4b73e68e11, State: Initialized, Role: FOLLOWER
I20260806 23:52:49.807967 27072 consensus_queue.cc:260] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } }
I20260806 23:52:49.808900 27072 ts_tablet_manager.cc:1428] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 23:52:49.809052 26693 heartbeater.cc:503] Master 127.10.225.190:34187 was elected leader, sending a full tablet report...
I20260806 23:52:49.979463 27079 raft_consensus.cc:482] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:52:49.979761 27079 raft_consensus.cc:504] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } }
W20260806 23:52:49.980376 27048 tablet.cc:2369] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:52:49.981144 27079 leader_election.cc:288] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5826edd2c6a24e9e8b08ea4b73e68e11 (127.10.225.129:42855), 5fef0017ee8846db96520a7765f528f8 (127.10.225.130:42623)
I20260806 23:52:49.985461 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.981601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:54789 (local address 127.10.225.129:42855)
0806 23:52:49.981739 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:49.981744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:49.981760 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:52:49.981819 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:49.981822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:49.981870 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:49.981954 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:49.982318 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.982990 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.984393 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.984914 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.985040 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.985095 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:49.985258 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 23:52:49.985335 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:52:49.985482 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.981445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:42855 (local address 127.10.225.131:54789)
0806 23:52:49.981576 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:49.981591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:49.981703 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:49.981995 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:49.981999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:49.982027 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:49.982165 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.982174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.983099 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.983102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.984267 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.984281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.985044 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.985047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.985136 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.985160 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:52:49.985237 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 23:52:49.985313 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":13}
I20260806 23:52:49.985826 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.981893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:35219 (local address 127.10.225.130:42623)
0806 23:52:49.982024 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:49.982028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:49.982285 (+ 257us) server_negotiation.cc:407] Connection header received
0806 23:52:49.982405 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:49.982408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:49.982468 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:49.982545 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:52:49.982775 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.983357 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.984672 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.985202 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.985312 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.985356 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:49.985613 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 23:52:49.985699 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:52:49.986203 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:49.981784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:42623 (local address 127.10.225.131:35219)
0806 23:52:49.982202 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:49.982217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:52:49.982313 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:52:49.982557 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:52:49.982559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:52:49.982572 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:52:49.982679 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.982684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.983468 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.983471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.984566 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:52:49.984575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:52:49.985328 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:52:49.985332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:52:49.985440 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:49.985468 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 23:52:49.985985 (+ 517us) client_negotiation.cc:240] Negotiation successful
0806 23:52:49.986064 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":126,"threads_started":1}
I20260806 23:52:49.986204 26634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43d50f077934651b373f291cadaf76a" candidate_uuid: "cfe06faac6af44c28647b8aa8444f0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" is_pre_election: true
I20260806 23:52:49.986557 26634 raft_consensus.cc:2454] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfe06faac6af44c28647b8aa8444f0f2 in term 0.
I20260806 23:52:49.986865 26811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43d50f077934651b373f291cadaf76a" candidate_uuid: "cfe06faac6af44c28647b8aa8444f0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fef0017ee8846db96520a7765f528f8" is_pre_election: true
W20260806 23:52:49.986937 26873 tablet.cc:2369] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:52:49.987169 26811 raft_consensus.cc:2454] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfe06faac6af44c28647b8aa8444f0f2 in term 0.
I20260806 23:52:49.987144 26898 leader_election.cc:302] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5826edd2c6a24e9e8b08ea4b73e68e11, cfe06faac6af44c28647b8aa8444f0f2; no voters:
I20260806 23:52:49.987457 27079 raft_consensus.cc:2781] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:52:49.987555 27079 raft_consensus.cc:482] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:52:49.987599 27079 raft_consensus.cc:3037] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:49.988667 27079 raft_consensus.cc:504] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } }
I20260806 23:52:49.989004 27079 leader_election.cc:288] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [CANDIDATE]: Term 1 election: Requested vote from peers 5826edd2c6a24e9e8b08ea4b73e68e11 (127.10.225.129:42855), 5fef0017ee8846db96520a7765f528f8 (127.10.225.130:42623)
I20260806 23:52:49.989380 26634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43d50f077934651b373f291cadaf76a" candidate_uuid: "cfe06faac6af44c28647b8aa8444f0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11"
I20260806 23:52:49.989512 26811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f43d50f077934651b373f291cadaf76a" candidate_uuid: "cfe06faac6af44c28647b8aa8444f0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fef0017ee8846db96520a7765f528f8"
I20260806 23:52:49.989529 26634 raft_consensus.cc:3037] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:49.989637 26811 raft_consensus.cc:3037] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:52:49.990620 26811 raft_consensus.cc:2454] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfe06faac6af44c28647b8aa8444f0f2 in term 1.
I20260806 23:52:49.990630 26634 raft_consensus.cc:2454] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfe06faac6af44c28647b8aa8444f0f2 in term 1.
I20260806 23:52:49.990886 26898 leader_election.cc:302] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fef0017ee8846db96520a7765f528f8, cfe06faac6af44c28647b8aa8444f0f2; no voters:
I20260806 23:52:49.991061 27079 raft_consensus.cc:2781] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:52:49.991273 27079 raft_consensus.cc:686] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 1 LEADER]: Becoming Leader. State: Replica: cfe06faac6af44c28647b8aa8444f0f2, State: Running, Role: LEADER
I20260806 23:52:49.991595 27079 consensus_queue.cc:237] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } }
I20260806 23:52:49.995193 26438 catalog_manager.cc:5495] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfe06faac6af44c28647b8aa8444f0f2 (127.10.225.131). New cstate: current_term: 1 leader_uuid: "cfe06faac6af44c28647b8aa8444f0f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfe06faac6af44c28647b8aa8444f0f2" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 } health_report { overall_health: UNKNOWN } } }
W20260806 23:52:50.000010 26695 tablet.cc:2369] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:52:50.023216 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:50.019391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.10.225.131:37813)
0806 23:52:50.019732 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 23:52:50.019736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:52:50.019752 (+ 16us) server_negotiation.cc:407] Connection header received
0806 23:52:50.019830 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:52:50.019833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:52:50.019894 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:52:50.019988 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:52:50.020275 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:50.020786 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:50.021658 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:52:50.021899 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:52:50.021987 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:52:50.022278 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:52:50.022728 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:52:50.022822 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:52:50.022978 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 23:52:50.023049 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 23:52:50.029501 26811 raft_consensus.cc:1261] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [term 1 FOLLOWER]: Refusing update from remote peer cfe06faac6af44c28647b8aa8444f0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:50.029502 26634 raft_consensus.cc:1261] T f43d50f077934651b373f291cadaf76a P 5826edd2c6a24e9e8b08ea4b73e68e11 [term 1 FOLLOWER]: Refusing update from remote peer cfe06faac6af44c28647b8aa8444f0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:52:50.030217 27079 consensus_queue.cc:1035] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5826edd2c6a24e9e8b08ea4b73e68e11" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:50.030442 27082 consensus_queue.cc:1035] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fef0017ee8846db96520a7765f528f8" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:52:50.039170 27086 mvcc.cc:205] Tried to move back new op lower bound from 7315703275632361472 to 7315703275488522240. Current Snapshot: MvccSnapshot[applied={T|T < 7315703275632361472}]
I20260806 23:52:50.048025 26438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:50.048203 26438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:52:50.050796 26438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84c508fdcc1747649d73b7600f06ef89: Sending DeleteTablet for 3 replicas of tablet f43d50f077934651b373f291cadaf76a
I20260806 23:52:50.051788 26954 tablet_service.cc:1514] Processing DeleteTablet for tablet f43d50f077934651b373f291cadaf76a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:50 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:51876
I20260806 23:52:50.052048 26783 tablet_service.cc:1514] Processing DeleteTablet for tablet f43d50f077934651b373f291cadaf76a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:50 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:54260
I20260806 23:52:50.052081 26604 tablet_service.cc:1514] Processing DeleteTablet for tablet f43d50f077934651b373f291cadaf76a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:52:50 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:54948
I20260806 23:52:50.052563 27096 tablet_replica.cc:330] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: stopping tablet replica
I20260806 23:52:50.052908 27096 raft_consensus.cc:2229] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:52:50.053025 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 26524
I20260806 23:52:50.062125 27099 tablet_replica.cc:330] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: stopping tablet replica
I20260806 23:52:50.062413 27096 raft_consensus.cc:2258] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:50.062752 27099 raft_consensus.cc:2229] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:52:50.063133 27099 raft_consensus.cc:2258] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:52:50.064256 27096 ts_tablet_manager.cc:1905] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:50.064813 27099 ts_tablet_manager.cc:1905] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:52:50.067137 27096 ts_tablet_manager.cc:1918] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:50.067246 27096 log.cc:1196] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/f43d50f077934651b373f291cadaf76a
I20260806 23:52:50.067579 27096 ts_tablet_manager.cc:1939] T f43d50f077934651b373f291cadaf76a P cfe06faac6af44c28647b8aa8444f0f2: Deleting consensus metadata
I20260806 23:52:50.067864 27099 ts_tablet_manager.cc:1918] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:52:50.068032 27099 log.cc:1196] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/f43d50f077934651b373f291cadaf76a
I20260806 23:52:50.068392 27099 ts_tablet_manager.cc:1939] T f43d50f077934651b373f291cadaf76a P 5fef0017ee8846db96520a7765f528f8: Deleting consensus metadata
I20260806 23:52:50.068976 26416 catalog_manager.cc:4841] TS cfe06faac6af44c28647b8aa8444f0f2 (127.10.225.131:37813): tablet f43d50f077934651b373f291cadaf76a (table test-table [id=3d5f05b76d494443b8a7a9c7392f31b9]) successfully deleted
W20260806 23:52:50.069715 26413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:42855 (error 108)
I20260806 23:52:50.070233 26413 catalog_manager.cc:4841] TS 5fef0017ee8846db96520a7765f528f8 (127.10.225.130:42623): tablet f43d50f077934651b373f291cadaf76a (table test-table [id=3d5f05b76d494443b8a7a9c7392f31b9]) successfully deleted
I20260806 23:52:50.070793 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 26700
I20260806 23:52:50.070920 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:52:50.070621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:42855 (local address 127.0.0.1:54962)
0806 23:52:50.070756 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 23:52:50.070821 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 23:52:50.103864 26413 catalog_manager.cc:4584] TS 5826edd2c6a24e9e8b08ea4b73e68e11 (127.10.225.129:42855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f43d50f077934651b373f291cadaf76a: Network error: Client connection negotiation failed: client connection to 127.10.225.129:42855: connect: Connection refused (error 111)
I20260806 23:52:50.116084 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 26879
I20260806 23:52:50.131009 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 26395
2026-08-06T23:52:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2723 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:52:50 dist-test-slave-nj21 krb5kdc[27107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:52:50 dist-test-slave-nj21 krb5kdc[27107](info): set up 2 sockets
Aug 06 23:52:50 dist-test-slave-nj21 krb5kdc[27107](info): commencing operation
krb5kdc: starting...
W20260806 23:52:51.306591 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:52:51 dist-test-slave-nj21 krb5kdc[27107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:52:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/rangeradmin_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/rangeradmin_127.10.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/rangerlookup_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/rangerlookup_127.10.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/HTTP_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/HTTP_127.10.225.148@KRBTEST.COM.keytab.
I20260806 23:52:51.496502 11142 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyweHpq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/postgres -l logfile start
2026-08-06 23:52:54.060 UTC [27151] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 23:52:54.060 UTC [27151] LOG: listening on IPv4 address "127.10.225.148", port 34791
2026-08-06 23:52:54.064 UTC [27151] LOG: listening on Unix socket "/tmp/.s.PGSQL.34791"
2026-08-06 23:52:54.094 UTC [27156] LOG: database system was shut down at 2026-08-06 23:52:52 UTC
2026-08-06 23:52:54.098 UTC [27151] LOG: database system is ready to accept connections
I20260806 23:52:55.146651 11142 mini_postgres.cc:96] Postgres bound to 34791
2026-08-06 23:52:55.155 UTC [27171] FATAL: database "slave" does not exist
127.10.225.148:34791 - accepting connections
I20260806 23:52:55.155998 11142 mini_ranger.cc:161] Starting Ranger...
I20260806 23:52:55.173686 11142 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 23:52:55.634509 11142 mini_ranger.cc:87] Created ranger Postgres database
I20260806 23:52:55.634641 11142 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-admin
2026-08-06 23:52:55,957 [I] DB FLAVOR :POSTGRES
2026-08-06 23:52:55,957 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 23:52:55,958 [I] Checking connection..
2026-08-06 23:52:55,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 23:52:56,239 [I] Checking connection passed.
2026-08-06 23:52:56,239 [I] --------- Verifying version history table ---------
2026-08-06 23:52:56,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:52:56,526 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 23:52:56,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:52:56,814 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 23:52:56,815 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 23:52:56,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 23:52:57.088 UTC [27253] WARNING: there is no transaction in progress
2026-08-06 23:52:57,097 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 23:52:57,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:52:57,380 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 23:52:57,380 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 23:52:57,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 23:52:57,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 23:52:57,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nj21.c.gcp-upstream.internal','N') ;"
2026-08-06 23:52:58,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 23:52:58,456 [I] Table x_portal_user does not exist in database ranger
2026-08-06 23:52:58,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 23:52:58,711 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 23:52:58,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 23:52:58,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 23:52:59,240 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 23:52:59,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 23:53:00.266 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.283 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.301 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.317 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.332 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.351 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.410 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.419 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.434 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.445 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.456 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.472 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.484 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.491 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.499 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:00.507 UTC [27409] WARNING: there is no transaction in progress
2026-08-06 23:53:01,250 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 23:53:01,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 23:53:01,526 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 23:53:01,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 23:53:01,790 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 23:53:01,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 23:53:02,066 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 23:53:02,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 23:53:02,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 23:53:02,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nj21.c.gcp-upstream.internal';"
2026-08-06 23:53:02,912 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 23:53:02,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 23:53:03,206 [I] DB_PATCHES have already been applied
I20260806 23:53:03.212711 11142 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyweHpq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 23:53:03.212792 11142 mini_ranger.cc:193] Using host: 127.10.225.148
I20260806 23:53:03.218714 11142 mini_ranger.cc:227] Ranger admin URL: http://127.10.225.148:35167
Aug 06, 2026 11:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 11:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 11:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 11:53:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 11:53:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 23:53:04.449501 11142 mini_ranger.cc:160] Time spent starting Ranger: real 9.294s user 0.001s sys 0.013s
Aug 06, 2026 11:53:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 11:53:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 11:53:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 11:53:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 11:53:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 23:53:28.260680 11142 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/rangerkms_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/rangerkms_127.10.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.225.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.225.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.225.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/HTTP_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.225.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/HTTP_127.10.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 23:53:28 dist-test-slave-nj21 krb5kdc[27107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060408, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 23:53:28.356863 11142 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 23:53:28.368183 11142 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 23:53:29.235716 11142 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 23:53:29.235843 11142 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-kms
I20260806 23:53:29.236063 11142 mini_ranger_kms.cc:227] Using postgres at 127.10.225.148:34791
2026-08-06 23:53:30,228 [I] DB FLAVOR :POSTGRES
2026-08-06 23:53:30,229 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 23:53:30,229 [I] Checking connection
2026-08-06 23:53:30,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 23:53:30,536 [I] connection success
2026-08-06 23:53:30,537 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 23:53:30,537 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 23:53:30,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 23:53:30,841 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 23:53:30,842 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 23:53:30,842 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 23:53:30,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:34791/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 23:53:31,126 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 23:53:31.659541 11142 mini_ranger_kms.cc:310] Created kms service
I20260806 23:53:31.882972 11142 mini_ranger_kms.cc:326] Created kudu user
I20260806 23:53:31.932333 11142 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 23:53:31 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060408, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:53:32.384503 11142 mini_ranger_kms.cc:384] Added ranger policy
I20260806 23:53:32.384670 11142 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 23:53:32.384689 11142 mini_ranger_kms.cc:240] Using host: 127.10.225.148
I20260806 23:53:32.390657 11142 mini_ranger_kms.cc:276] Ranger KMS PID: 27659
I20260806 23:53:32.390782 11142 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.225.148:48739
Aug 06, 2026 11:53:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:53:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:53:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 11:53:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 11:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 11:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 23:53:34 dist-test-slave-nj21 krb5kdc[27107](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060414, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.225.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:53:34.956640 11142 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.600s user 0.030s sys 0.659s
I20260806 23:53:34.956786 11142 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 23:53:34.956899 11142 mini_ranger_kms.cc:401] 127.10.225.148:48739/kms/v1/keys
Aug 06 23:53:35 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060414, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.225.148@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
Aug 06 23:53:35 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060414, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.225.148@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
Aug 06 23:53:38 dist-test-slave-nj21 krb5kdc[27107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:53:38.319798 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:46155
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:46155
--ranger_config_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:48739/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:53:38.469596 27721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:53:38.469925 27721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:53:38.469988 27721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:53:38.474925 27721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:53:38.475021 27721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:53:38.475051 27721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:53:38.475075 27721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:53:38.475104 27721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:53:38.484072 27721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.225.148:48739/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:46155
--ranger_config_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:53:38.485847 27721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:53:38.487344 27721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:53:38.498886 27733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:53:38.499003 27735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:53:38.498955 27737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:53:38.499228 27721 server_base.cc:1034] running on GCE node
I20260806 23:53:38.500180 27721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:53:38.501812 27721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:53:38.503008 27721 hybrid_clock.cc:648] HybridClock initialized: now 1786060418502990 us; error 35 us; skew 500 ppm
Aug 06 23:53:38 dist-test-slave-nj21 krb5kdc[27107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060418, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:53:38.509827 27721 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:53:38.510896 27721 webserver.cc:460] Webserver started at http://127.10.225.190:34251/ using document root <none> and password file <none>
I20260806 23:53:38.511471 27721 fs_manager.cc:359] Metadata directory not provided
I20260806 23:53:38.511528 27721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:53:38.511761 27721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:53:38 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060418, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:53:38.565086 27721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "1547dde5eeee4936a438a7ea31daf546"
format_stamp: "Formatted at 2026-08-06 23:53:38 on dist-test-slave-nj21"
server_key: "93f3bf6a7f6b01c084b93561d4acd7fa"
server_key_iv: "35e43ebdeb2bc74a3bdb589122ad040b"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:38.565686 27721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "1547dde5eeee4936a438a7ea31daf546"
format_stamp: "Formatted at 2026-08-06 23:53:38 on dist-test-slave-nj21"
server_key: "93f3bf6a7f6b01c084b93561d4acd7fa"
server_key_iv: "35e43ebdeb2bc74a3bdb589122ad040b"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:38.569679 27721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:53:38.585444 27748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:38.586839 27721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:53:38.586975 27721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "1547dde5eeee4936a438a7ea31daf546"
format_stamp: "Formatted at 2026-08-06 23:53:38 on dist-test-slave-nj21"
server_key: "93f3bf6a7f6b01c084b93561d4acd7fa"
server_key_iv: "35e43ebdeb2bc74a3bdb589122ad040b"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:38.587108 27721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:53:38.615877 27721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:53:38.619809 27721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:53:38.620028 27721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:53:38.631649 27721 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:46155
I20260806 23:53:38.631842 27817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:46155 every 8 connection(s)
I20260806 23:53:38.632812 27721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:53:38.636021 27818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:53:38.641155 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 27721
I20260806 23:53:38.641396 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:53:38.641894 11142 external_mini_cluster.cc:1428] Setting key b9d9954055412beaae931f4bfe86fdd0
I20260806 23:53:38.644413 27818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546: Bootstrap starting.
I20260806 23:53:38.647722 27818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546: Neither blocks nor log segments found. Creating new log.
I20260806 23:53:38.648888 27818 log.cc:824] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546: Log is configured to *not* fsync() on all Append() calls
I20260806 23:53:38.651418 27818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546: No bootstrap required, opened a new log
Aug 06 23:53:38 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:53:38.655783 27818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547dde5eeee4936a438a7ea31daf546" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46155 } }
I20260806 23:53:38.656088 27818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:53:38.656165 27818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1547dde5eeee4936a438a7ea31daf546, State: Initialized, Role: FOLLOWER
I20260806 23:53:38.656797 27818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547dde5eeee4936a438a7ea31daf546" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46155 } }
I20260806 23:53:38.656991 27818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:53:38.657068 27818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:53:38.657182 27818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:53:38.658666 27818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547dde5eeee4936a438a7ea31daf546" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46155 } }
I20260806 23:53:38.659070 27818 leader_election.cc:302] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1547dde5eeee4936a438a7ea31daf546; no voters:
I20260806 23:53:38.659473 27818 leader_election.cc:288] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:53:38.659658 27825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:38.643796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.10.225.190:46155)
0806 23:53:38.644527 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:38.644547 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:38.644610 (+ 63us) server_negotiation.cc:407] Connection header received
0806 23:53:38.645637 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:38.645659 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:38.646069 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:38.646427 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:53:38.647244 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:38.648519 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:38.649421 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:38.649994 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:38.650153 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:38.652685 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:53:38.652693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:53:38.652712 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:53:38.652745 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:53:38.655453 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:38.656055 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:38.656061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:38.656068 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:38.656265 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:38.656814 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:38.656818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:38.656820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:38.657530 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:53:38.657756 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:38.658219 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 23:53:38.658637 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":206,"threads_started":1}
I20260806 23:53:38.659593 27829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:53:38.660661 27829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [term 1 LEADER]: Becoming Leader. State: Replica: 1547dde5eeee4936a438a7ea31daf546, State: Running, Role: LEADER
I20260806 23:53:38.661022 27818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:53:38.661111 27829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547dde5eeee4936a438a7ea31daf546" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46155 } }
I20260806 23:53:38.663318 27830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1547dde5eeee4936a438a7ea31daf546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547dde5eeee4936a438a7ea31daf546" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46155 } } }
I20260806 23:53:38.663553 27830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [sys.catalog]: This master's current role is: LEADER
I20260806 23:53:38.665021 27831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1547dde5eeee4936a438a7ea31daf546. Latest consensus state: current_term: 1 leader_uuid: "1547dde5eeee4936a438a7ea31daf546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1547dde5eeee4936a438a7ea31daf546" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 46155 } } }
I20260806 23:53:38.665126 27845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:53:38.665138 27831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546 [sys.catalog]: This master's current role is: LEADER
I20260806 23:53:38.665215 27818 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 23:53:38.672699 27845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:53:38.681912 27845 catalog_manager.cc:1347] Generated new cluster ID: 37d787cf6c9c494187408dde0d634e20
I20260806 23:53:38.682000 27845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:53:38.724869 27845 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:53:38.726652 27845 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:53:38.737782 27845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546: Generated new TSK 0
I20260806 23:53:38.738791 27845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:53:39.571419 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:48739/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46155
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:53:39.736958 27887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:53:39.737233 27887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:53:39.737299 27887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:53:39.742722 27887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:53:39.742839 27887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:53:39.742933 27887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:53:39.750394 27887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.225.148:48739/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:53:39.751559 27887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:53:39.752676 27887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:53:39.767011 27893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:53:39.767280 27887 server_base.cc:1034] running on GCE node
W20260806 23:53:39.767395 27892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:53:39.767498 27896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:53:39.768082 27887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:53:39.768973 27887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:53:39.770182 27887 hybrid_clock.cc:648] HybridClock initialized: now 1786060419770148 us; error 48 us; skew 500 ppm
Aug 06 23:53:39 dist-test-slave-nj21 krb5kdc[27107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060419, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:53:39.774030 27887 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:53:39.775164 27887 webserver.cc:460] Webserver started at http://127.10.225.129:38243/ using document root <none> and password file <none>
I20260806 23:53:39.775844 27887 fs_manager.cc:359] Metadata directory not provided
I20260806 23:53:39.775990 27887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:53:39.776247 27887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:53:39 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060419, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:53:39.821161 27887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "293be72579c14d4289f6c56e6867181f"
format_stamp: "Formatted at 2026-08-06 23:53:39 on dist-test-slave-nj21"
server_key: "c8e84d8541011a5f50e05765bb8c3a0a"
server_key_iv: "c505227245ee1f15d974fe542d991d70"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:39.828235 27887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "293be72579c14d4289f6c56e6867181f"
format_stamp: "Formatted at 2026-08-06 23:53:39 on dist-test-slave-nj21"
server_key: "c8e84d8541011a5f50e05765bb8c3a0a"
server_key_iv: "c505227245ee1f15d974fe542d991d70"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:39.864720 27887 fs_manager.cc:691] Time spent creating directory manager: real 0.034s user 0.024s sys 0.000s
I20260806 23:53:39.972093 27912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:39.973546 27887 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260806 23:53:39.973680 27887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "293be72579c14d4289f6c56e6867181f"
format_stamp: "Formatted at 2026-08-06 23:53:39 on dist-test-slave-nj21"
server_key: "c8e84d8541011a5f50e05765bb8c3a0a"
server_key_iv: "c505227245ee1f15d974fe542d991d70"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:39.973817 27887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:53:39.987756 27887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:53:39.991457 27887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:53:39.991765 27887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:53:39.992674 27887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:53:39.993901 27887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:53:39.994006 27887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:39.994138 27887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:53:39.994218 27887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:40.013849 27887 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:34221
I20260806 23:53:40.015196 27887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:53:40.021297 28070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:34221 every 8 connection(s)
I20260806 23:53:40.021447 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 27887
I20260806 23:53:40.021587 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:53:40.021934 11142 external_mini_cluster.cc:1428] Setting key e2c267af6b2b30757aca7d4f91a61020
Aug 06 23:53:40 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060419, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:53:40.042243 28074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.019095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46155 (local address 127.10.225.129:55761)
0806 23:53:40.022449 (+ 3354us) negotiation.cc:107] Waiting for socket to connect
0806 23:53:40.026786 (+ 4337us) client_negotiation.cc:174] Beginning negotiation
0806 23:53:40.028385 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:53:40.029525 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:53:40.029537 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:53:40.029932 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:53:40.030659 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.030678 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.032148 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.032151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.033467 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.033480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.034163 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.034167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.034392 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.035186 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:53:40.035209 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:53:40.037056 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:53:40.039377 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:53:40.039384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:53:40.039402 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:53:40.039714 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:53:40.040049 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:53:40.040053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:53:40.040055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:53:40.040199 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:53:40.040609 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:53:40.040616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:53:40.040901 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 23:53:40.041091 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 23:53:40.041397 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2724,"thread_start_us":172,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:53:40.045575 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.020480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:55761 (local address 127.10.225.190:46155)
0806 23:53:40.022191 (+ 1711us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:40.022196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:40.028830 (+ 6634us) server_negotiation.cc:407] Connection header received
0806 23:53:40.028886 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:40.028889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:40.028955 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:40.029040 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:53:40.031097 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.031992 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.033611 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.033863 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.033967 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.037295 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:53:40.037300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:53:40.037303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:53:40.037333 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:53:40.039224 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:40.039843 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:40.039847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:40.039849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:40.039947 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:40.040307 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:40.040311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:40.040313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:40.040506 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:53:40.040589 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:40.043135 (+ 2546us) server_negotiation.cc:299] Negotiation successful
0806 23:53:40.043269 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1615,"thread_start_us":131,"threads_started":1}
I20260806 23:53:40.046907 28071 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46155
I20260806 23:53:40.047181 28071 heartbeater.cc:464] Registering TS with master...
I20260806 23:53:40.047874 28071 heartbeater.cc:511] Master 127.10.225.190:46155 requested a full tablet report, sending...
I20260806 23:53:40.049865 27768 ts_manager.cc:194] Registered new tserver with Master: 293be72579c14d4289f6c56e6867181f (127.10.225.129:34221)
I20260806 23:53:40.051504 27768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:55761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:53:40.067574 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:48739/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46155
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:53:40 dist-test-slave-nj21 krb5kdc[27107](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060420, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 23:53:40.260970 28082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:53:40.261523 28082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:53:40.261620 28082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:53:40.268483 28082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:53:40.268738 28082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:53:40.268930 28082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:53:40.278831 28082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.225.148:48739/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:53:40.280409 28082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:53:40.281845 28082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:53:40.294107 28093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:53:40.294252 28095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:53:40.294107 28092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:53:40.295286 28082 server_base.cc:1034] running on GCE node
I20260806 23:53:40.295866 28082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:53:40.296834 28082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:53:40.298029 28082 hybrid_clock.cc:648] HybridClock initialized: now 1786060420297965 us; error 82 us; skew 500 ppm
Aug 06 23:53:40 dist-test-slave-nj21 krb5kdc[27107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060420, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:53:40.301973 28082 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:53:40.303118 28082 webserver.cc:460] Webserver started at http://127.10.225.130:35685/ using document root <none> and password file <none>
I20260806 23:53:40.303844 28082 fs_manager.cc:359] Metadata directory not provided
I20260806 23:53:40.303942 28082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:53:40.304178 28082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:53:40 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060420, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:53:40.323328 28082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "dc38ca701e594daa9c8c80eef3d4f6db"
format_stamp: "Formatted at 2026-08-06 23:53:40 on dist-test-slave-nj21"
server_key: "7bc9cbd0da9d89ff3692b5e27343e70d"
server_key_iv: "37ad05f01472fcb3cb06e3068aefe53d"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:40.324559 28082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "dc38ca701e594daa9c8c80eef3d4f6db"
format_stamp: "Formatted at 2026-08-06 23:53:40 on dist-test-slave-nj21"
server_key: "7bc9cbd0da9d89ff3692b5e27343e70d"
server_key_iv: "37ad05f01472fcb3cb06e3068aefe53d"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:40.331187 28082 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 23:53:40.368196 28103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:40.370163 28082 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.002s sys 0.000s
I20260806 23:53:40.370327 28082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "dc38ca701e594daa9c8c80eef3d4f6db"
format_stamp: "Formatted at 2026-08-06 23:53:40 on dist-test-slave-nj21"
server_key: "7bc9cbd0da9d89ff3692b5e27343e70d"
server_key_iv: "37ad05f01472fcb3cb06e3068aefe53d"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:40.370512 28082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:53:40.387383 28082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:53:40.392310 28082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:53:40.392539 28082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:53:40.393420 28082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:53:40.394579 28082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:53:40.394640 28082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:40.394697 28082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:53:40.394718 28082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:40.417408 28082 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:37911
I20260806 23:53:40.418825 28082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:53:40.423862 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 28082
I20260806 23:53:40.424003 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:53:40.424325 11142 external_mini_cluster.cc:1428] Setting key 51e3e1faf0b7a3d51cb89fc85969cd27
I20260806 23:53:40.420782 28263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:37911 every 8 connection(s)
Aug 06 23:53:40 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060420, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:53:40.443368 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.424002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:57997 (local address 127.10.225.190:46155)
0806 23:53:40.424164 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:40.424169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:40.426657 (+ 2488us) server_negotiation.cc:407] Connection header received
0806 23:53:40.428210 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:40.428213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:40.428267 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:40.428324 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:53:40.430143 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.430939 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.432820 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.433161 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.433247 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.438009 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:53:40.438014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:53:40.438017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:53:40.438050 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:53:40.440399 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:40.441360 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:40.441364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:40.441366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:40.441474 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:40.441945 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:40.441949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:40.441951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:40.442205 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:53:40.442291 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:40.443026 (+ 735us) server_negotiation.cc:299] Negotiation successful
0806 23:53:40.443193 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 23:53:40.445580 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.424415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46155 (local address 127.10.225.130:57997)
0806 23:53:40.426490 (+ 2075us) negotiation.cc:107] Waiting for socket to connect
0806 23:53:40.426551 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 23:53:40.427949 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:53:40.428745 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:53:40.428755 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:53:40.429126 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:53:40.429952 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.429977 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.431102 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.431105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.432666 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.432678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.433474 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.433478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.433762 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.435848 (+ 2086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:53:40.435871 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:53:40.437748 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:53:40.440761 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:53:40.440771 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:53:40.440795 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:53:40.441220 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:53:40.441601 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:53:40.441604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:53:40.441606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:53:40.441809 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:53:40.442364 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:53:40.442371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:53:40.442719 (+ 348us) client_negotiation.cc:769] Sending connection context
0806 23:53:40.443946 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0806 23:53:40.444333 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1789,"thread_start_us":306,"threads_started":1}
I20260806 23:53:40.450714 28264 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46155
I20260806 23:53:40.451020 28264 heartbeater.cc:464] Registering TS with master...
I20260806 23:53:40.451805 28264 heartbeater.cc:511] Master 127.10.225.190:46155 requested a full tablet report, sending...
I20260806 23:53:40.453044 27768 ts_manager.cc:194] Registered new tserver with Master: dc38ca701e594daa9c8c80eef3d4f6db (127.10.225.130:37911)
I20260806 23:53:40.453686 27768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:57997
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:53:40.491793 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:48739/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46155
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:53:40.684279 28273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:53:40.684674 28273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:53:40.685351 28273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:53:40.691859 28273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:53:40.692003 28273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:53:40.692139 28273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:53:40.700568 28273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.225.148:48739/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:46155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:53:40.702262 28273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:53:40.703730 28273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:53:40.718604 28281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:53:40.718604 28283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:53:40.720009 28273 server_base.cc:1034] running on GCE node
W20260806 23:53:40.719830 28280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:53:40.720791 28273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:53:40.721686 28273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:53:40.722849 28273 hybrid_clock.cc:648] HybridClock initialized: now 1786060420722786 us; error 78 us; skew 500 ppm
Aug 06 23:53:40 dist-test-slave-nj21 krb5kdc[27107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060420, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:53:40.727414 28273 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:53:40.728881 28273 webserver.cc:460] Webserver started at http://127.10.225.131:33105/ using document root <none> and password file <none>
I20260806 23:53:40.729679 28273 fs_manager.cc:359] Metadata directory not provided
I20260806 23:53:40.729822 28273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:53:40.730113 28273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:53:40 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060420, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:53:40.751803 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "e9038a15c04c4a8fb32d75162d4dd871"
format_stamp: "Formatted at 2026-08-06 23:53:40 on dist-test-slave-nj21"
server_key: "29bf7c09d0a39a8cd5d9077d20492e34"
server_key_iv: "4248bd078897be52f6997847ee134975"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:40.753463 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "e9038a15c04c4a8fb32d75162d4dd871"
format_stamp: "Formatted at 2026-08-06 23:53:40 on dist-test-slave-nj21"
server_key: "29bf7c09d0a39a8cd5d9077d20492e34"
server_key_iv: "4248bd078897be52f6997847ee134975"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:40.758574 28273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 23:53:40.786748 28291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:40.788664 28273 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260806 23:53:40.788818 28273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "e9038a15c04c4a8fb32d75162d4dd871"
format_stamp: "Formatted at 2026-08-06 23:53:40 on dist-test-slave-nj21"
server_key: "29bf7c09d0a39a8cd5d9077d20492e34"
server_key_iv: "4248bd078897be52f6997847ee134975"
server_key_version: "kuduclusterkey@0"
I20260806 23:53:40.788978 28273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 06 23:53:40 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060420, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:53:40.834939 28273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:53:40.840610 28273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:53:40.841065 28273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:53:40.842384 28273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:53:40.843731 28273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:53:40.843854 28273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:40.843976 28273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:53:40.844053 28273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:53:40.874184 28273 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:42355
I20260806 23:53:40.875909 28273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:53:40.881070 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 28273
I20260806 23:53:40.881217 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:53:40.881537 11142 external_mini_cluster.cc:1428] Setting key 03955623fa89b0a6fff32d570a63041e
I20260806 23:53:40.884159 28447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:42355 every 8 connection(s)
Aug 06 23:53:40 dist-test-slave-nj21 krb5kdc[27107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060420, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:53:40.903330 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.888460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:33917 (local address 127.10.225.190:46155)
0806 23:53:40.888580 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:40.888584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:40.888598 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:53:40.889431 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:40.889435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:40.889512 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:40.889584 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:53:40.891164 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.891943 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.893539 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.893852 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.893941 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.899068 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:53:40.899072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:53:40.899074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:53:40.899100 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:53:40.900704 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:40.901367 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:40.901371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:40.901373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:40.901466 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:40.901925 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:40.901929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:40.901931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:40.902170 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:53:40.902254 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:40.903023 (+ 769us) server_negotiation.cc:299] Negotiation successful
0806 23:53:40.903156 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:53:40.905520 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.887164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:46155 (local address 127.10.225.131:33917)
0806 23:53:40.888032 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0806 23:53:40.888095 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 23:53:40.889178 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:53:40.889790 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:53:40.889800 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:53:40.890178 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:53:40.891029 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.891054 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.892073 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.892077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.893395 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.893407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.894169 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.894173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.894388 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.897153 (+ 2765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:53:40.897176 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:53:40.898865 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:53:40.900853 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:53:40.900860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:53:40.900881 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:53:40.901248 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:53:40.901583 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:53:40.901587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:53:40.901588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:53:40.901791 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:53:40.902347 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:53:40.902355 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:53:40.902713 (+ 358us) client_negotiation.cc:769] Sending connection context
0806 23:53:40.903957 (+ 1244us) client_negotiation.cc:240] Negotiation successful
0806 23:53:40.904344 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":171,"threads_started":1}
I20260806 23:53:40.907171 28448 heartbeater.cc:347] Connected to a master server at 127.10.225.190:46155
I20260806 23:53:40.907423 28448 heartbeater.cc:464] Registering TS with master...
I20260806 23:53:40.908159 28448 heartbeater.cc:511] Master 127.10.225.190:46155 requested a full tablet report, sending...
I20260806 23:53:40.909276 27768 ts_manager.cc:194] Registered new tserver with Master: e9038a15c04c4a8fb32d75162d4dd871 (127.10.225.131:42355)
I20260806 23:53:40.910033 27768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:33917
I20260806 23:53:40.921085 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:53:40.935089 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.924171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.10.225.190:46155)
0806 23:53:40.924309 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:40.924314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:40.924802 (+ 488us) server_negotiation.cc:407] Connection header received
0806 23:53:40.924993 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:40.924996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:40.925046 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:40.925096 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:53:40.926659 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.927491 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.928796 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.929139 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.929222 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.931221 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:53:40.931225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:53:40.931228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:53:40.931256 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:53:40.932993 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:40.933497 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:40.933501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:40.933503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:40.933591 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:53:40.933945 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:53:40.933949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:53:40.933951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:53:40.934196 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:53:40.934305 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:40.934684 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 23:53:40.934815 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:53:40.939910 27768 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:53:40.945184 27768 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:53:40.976934 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.962940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:42355 (local address 127.0.0.1:42990)
0806 23:53:40.965521 (+ 2581us) negotiation.cc:107] Waiting for socket to connect
0806 23:53:40.965534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:53:40.965632 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:53:40.967802 (+ 2170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:53:40.967809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:53:40.967837 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:53:40.967994 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.968001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.969550 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.969554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.972457 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.972469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.976306 (+ 3837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.976310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.976455 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.976510 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 23:53:40.976612 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 23:53:40.976761 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2522,"thread_start_us":240,"threads_started":1}
I20260806 23:53:40.978453 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.963612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.10.225.131:42355)
0806 23:53:40.964971 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:40.964989 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:40.966071 (+ 1082us) server_negotiation.cc:407] Connection header received
0806 23:53:40.966158 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:40.966164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:40.966406 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:40.966603 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:53:40.968195 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.969384 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.972657 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.973976 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.977939 (+ 3963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.978107 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:40.978169 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 23:53:40.978282 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":237,"threads_started":1}
I20260806 23:53:40.982086 28343 tablet_service.cc:1467] Processing CreateTablet for tablet 13d6d62076a44fb280ffa7ea65572930 (DEFAULT_TABLE table=test-table [id=adffb99dcd5b40bd90c61f1bf783e49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:53:40.983731 28343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d6d62076a44fb280ffa7ea65572930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:53:40.985250 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.964910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:34221 (local address 127.0.0.1:45576)
0806 23:53:40.965842 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0806 23:53:40.965854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:53:40.965960 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:53:40.979507 (+ 13547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:53:40.979511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:53:40.979532 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:53:40.979664 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.979690 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.982086 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.982090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.983405 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.983417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.984812 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.984816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.984922 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.984955 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 23:53:40.985009 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 23:53:40.985090 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":155,"threads_started":1}
I20260806 23:53:40.987213 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.965097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.10.225.129:34221)
0806 23:53:40.978981 (+ 13884us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:40.978988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:40.979017 (+ 29us) server_negotiation.cc:407] Connection header received
0806 23:53:40.979100 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:40.979106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:40.979329 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:40.980734 (+ 1405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:53:40.980900 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.981943 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.983662 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.984656 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.986771 (+ 2115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.986912 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:40.986966 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 23:53:40.987064 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13765,"thread_start_us":164,"threads_started":1}
I20260806 23:53:40.990742 27955 tablet_service.cc:1467] Processing CreateTablet for tablet 13d6d62076a44fb280ffa7ea65572930 (DEFAULT_TABLE table=test-table [id=adffb99dcd5b40bd90c61f1bf783e49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:53:40.992201 27955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d6d62076a44fb280ffa7ea65572930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:53:40.998473 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.960553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:37911 (local address 127.0.0.1:36978)
0806 23:53:40.964389 (+ 3836us) negotiation.cc:107] Waiting for socket to connect
0806 23:53:40.964455 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 23:53:40.964617 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:53:40.976053 (+ 11436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:53:40.976057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:53:40.976081 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:53:40.976222 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.976229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.992534 (+ 16305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.992538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.993743 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:40.993755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.997965 (+ 4210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.997970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:40.998101 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.998137 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 23:53:40.998196 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 23:53:40.998288 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3616,"thread_start_us":123,"threads_started":1}
I20260806 23:53:40.999292 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:40.961614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.10.225.130:37911)
0806 23:53:40.974422 (+ 12808us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:40.974445 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:40.974475 (+ 30us) server_negotiation.cc:407] Connection header received
0806 23:53:40.974571 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:40.974577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:40.975888 (+ 1311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:40.976078 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:53:40.978143 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.992361 (+ 14218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.993920 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:40.995820 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:40.998825 (+ 3005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:40.998977 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:40.999035 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 23:53:40.999143 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12654,"thread_start_us":233,"threads_started":1}
I20260806 23:53:41.002566 28172 tablet_service.cc:1467] Processing CreateTablet for tablet 13d6d62076a44fb280ffa7ea65572930 (DEFAULT_TABLE table=test-table [id=adffb99dcd5b40bd90c61f1bf783e49b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:53:41.003785 28172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d6d62076a44fb280ffa7ea65572930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:53:41.007913 28473 tablet_bootstrap.cc:492] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f: Bootstrap starting.
I20260806 23:53:41.008141 28472 tablet_bootstrap.cc:492] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: Bootstrap starting.
I20260806 23:53:41.010771 28473 tablet_bootstrap.cc:654] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f: Neither blocks nor log segments found. Creating new log.
I20260806 23:53:41.011193 28472 tablet_bootstrap.cc:654] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: Neither blocks nor log segments found. Creating new log.
I20260806 23:53:41.011912 28473 log.cc:824] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f: Log is configured to *not* fsync() on all Append() calls
I20260806 23:53:41.012372 28472 log.cc:824] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: Log is configured to *not* fsync() on all Append() calls
I20260806 23:53:41.020239 28474 tablet_bootstrap.cc:492] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: Bootstrap starting.
I20260806 23:53:41.022080 28472 tablet_bootstrap.cc:492] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: No bootstrap required, opened a new log
I20260806 23:53:41.022336 28472 ts_tablet_manager.cc:1397] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20260806 23:53:41.023265 28474 tablet_bootstrap.cc:654] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: Neither blocks nor log segments found. Creating new log.
I20260806 23:53:41.024358 28474 log.cc:824] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: Log is configured to *not* fsync() on all Append() calls
I20260806 23:53:41.026928 28474 tablet_bootstrap.cc:492] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: No bootstrap required, opened a new log
I20260806 23:53:41.027143 28474 ts_tablet_manager.cc:1397] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 23:53:41.027133 28472 raft_consensus.cc:348] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } }
I20260806 23:53:41.027438 28472 raft_consensus.cc:374] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:53:41.027499 28472 raft_consensus.cc:729] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9038a15c04c4a8fb32d75162d4dd871, State: Initialized, Role: FOLLOWER
I20260806 23:53:41.037751 28473 tablet_bootstrap.cc:492] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f: No bootstrap required, opened a new log
I20260806 23:53:41.038000 28473 ts_tablet_manager.cc:1397] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20260806 23:53:41.040983 28474 raft_consensus.cc:348] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } }
I20260806 23:53:41.041329 28474 raft_consensus.cc:374] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:53:41.041393 28474 raft_consensus.cc:729] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc38ca701e594daa9c8c80eef3d4f6db, State: Initialized, Role: FOLLOWER
I20260806 23:53:41.041961 28474 consensus_queue.cc:260] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } }
I20260806 23:53:41.042142 28473 raft_consensus.cc:348] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } }
I20260806 23:53:41.042418 28473 raft_consensus.cc:374] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:53:41.042483 28473 raft_consensus.cc:729] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293be72579c14d4289f6c56e6867181f, State: Initialized, Role: FOLLOWER
I20260806 23:53:41.042944 28473 consensus_queue.cc:260] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } }
I20260806 23:53:41.043398 28474 ts_tablet_manager.cc:1428] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20260806 23:53:41.043913 28473 ts_tablet_manager.cc:1428] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:53:41.044234 28071 heartbeater.cc:503] Master 127.10.225.190:46155 was elected leader, sending a full tablet report...
I20260806 23:53:41.044852 28472 consensus_queue.cc:260] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } }
I20260806 23:53:41.045897 28472 ts_tablet_manager.cc:1428] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20260806 23:53:41.046234 28448 heartbeater.cc:503] Master 127.10.225.190:46155 was elected leader, sending a full tablet report...
I20260806 23:53:41.049000 28264 heartbeater.cc:503] Master 127.10.225.190:46155 was elected leader, sending a full tablet report...
I20260806 23:53:41.052445 28480 raft_consensus.cc:482] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:53:41.052656 28480 raft_consensus.cc:504] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } }
I20260806 23:53:41.054051 28480 leader_election.cc:288] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9038a15c04c4a8fb32d75162d4dd871 (127.10.225.131:42355), 293be72579c14d4289f6c56e6867181f (127.10.225.129:34221)
I20260806 23:53:41.062906 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:41.054683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:54565 (local address 127.10.225.129:34221)
0806 23:53:41.054879 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:41.054884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:41.056901 (+ 2017us) server_negotiation.cc:407] Connection header received
0806 23:53:41.056954 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:41.056957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:41.057013 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:41.057081 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:53:41.057583 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.058358 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.061514 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.062149 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.062246 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:41.062304 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:41.062654 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 23:53:41.062746 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:53:41.063378 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:41.054305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:34221 (local address 127.10.225.130:54565)
0806 23:53:41.055861 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0806 23:53:41.055874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:53:41.055983 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:53:41.057194 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:53:41.057197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:53:41.057215 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:53:41.057447 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:41.057455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.058500 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.058504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:41.059925 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:41.059937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.062414 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.062418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:41.062519 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:41.062543 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:53:41.063130 (+ 587us) client_negotiation.cc:240] Negotiation successful
0806 23:53:41.063214 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1492,"thread_start_us":135,"threads_started":1}
I20260806 23:53:41.064111 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d6d62076a44fb280ffa7ea65572930" candidate_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "293be72579c14d4289f6c56e6867181f" is_pre_election: true
I20260806 23:53:41.064481 27999 raft_consensus.cc:2454] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc38ca701e594daa9c8c80eef3d4f6db in term 0.
I20260806 23:53:41.065053 28105 leader_election.cc:302] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293be72579c14d4289f6c56e6867181f, dc38ca701e594daa9c8c80eef3d4f6db; no voters:
I20260806 23:53:41.065399 28480 raft_consensus.cc:2781] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:53:41.065487 28480 raft_consensus.cc:482] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:53:41.065531 28480 raft_consensus.cc:3037] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:53:41.067086 28480 raft_consensus.cc:504] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } }
I20260806 23:53:41.068024 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d6d62076a44fb280ffa7ea65572930" candidate_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "293be72579c14d4289f6c56e6867181f"
I20260806 23:53:41.068171 27999 raft_consensus.cc:3037] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:53:41.069819 27999 raft_consensus.cc:2454] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc38ca701e594daa9c8c80eef3d4f6db in term 1.
I20260806 23:53:41.070202 28105 leader_election.cc:302] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293be72579c14d4289f6c56e6867181f, dc38ca701e594daa9c8c80eef3d4f6db; no voters:
I20260806 23:53:41.070533 28480 leader_election.cc:288] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [CANDIDATE]: Term 1 election: Requested vote from peers e9038a15c04c4a8fb32d75162d4dd871 (127.10.225.131:42355), 293be72579c14d4289f6c56e6867181f (127.10.225.129:34221)
I20260806 23:53:41.070639 28480 raft_consensus.cc:2781] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:53:41.070713 28480 raft_consensus.cc:686] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 1 LEADER]: Becoming Leader. State: Replica: dc38ca701e594daa9c8c80eef3d4f6db, State: Running, Role: LEADER
I20260806 23:53:41.071118 28480 consensus_queue.cc:237] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } }
I20260806 23:53:41.074712 27768 catalog_manager.cc:5495] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db reported cstate change: term changed from 0 to 1, leader changed from <none> to dc38ca701e594daa9c8c80eef3d4f6db (127.10.225.130). New cstate: current_term: 1 leader_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 } health_report { overall_health: UNKNOWN } } }
I20260806 23:53:41.078441 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:41.054671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:41053 (local address 127.10.225.131:42355)
0806 23:53:41.054879 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:41.054884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:41.055172 (+ 288us) server_negotiation.cc:407] Connection header received
0806 23:53:41.055359 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:41.055362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:41.055409 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:41.055455 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:53:41.055960 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.056737 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.076255 (+ 19518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.077025 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.077170 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:41.077228 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:41.078175 (+ 947us) server_negotiation.cc:299] Negotiation successful
0806 23:53:41.078278 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 23:53:41.079123 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:41.054290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:42355 (local address 127.10.225.130:41053)
0806 23:53:41.055084 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0806 23:53:41.055099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:53:41.055244 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:53:41.055597 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:53:41.055600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:53:41.055632 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:53:41.055815 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:41.055825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.060075 (+ 4250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.060078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:41.061355 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:53:41.061367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.077805 (+ 16438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.077810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:53:41.078002 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:41.078029 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 23:53:41.078823 (+ 794us) client_negotiation.cc:240] Negotiation successful
0806 23:53:41.078921 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":140,"threads_started":1}
I20260806 23:53:41.080251 28364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d6d62076a44fb280ffa7ea65572930" candidate_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9038a15c04c4a8fb32d75162d4dd871" is_pre_election: true
I20260806 23:53:41.080613 28364 raft_consensus.cc:2454] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc38ca701e594daa9c8c80eef3d4f6db in term 0.
I20260806 23:53:41.081086 28365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d6d62076a44fb280ffa7ea65572930" candidate_uuid: "dc38ca701e594daa9c8c80eef3d4f6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9038a15c04c4a8fb32d75162d4dd871"
I20260806 23:53:41.081271 28365 raft_consensus.cc:3037] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:53:41.082965 28365 raft_consensus.cc:2454] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc38ca701e594daa9c8c80eef3d4f6db in term 1.
I20260806 23:53:41.096256 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:41.091479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36984 (local address 127.10.225.130:37911)
0806 23:53:41.091604 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 23:53:41.091609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:53:41.091626 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:53:41.091821 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:53:41.091824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:53:41.091905 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:53:41.092040 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:53:41.092318 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.093079 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.094312 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:53:41.094645 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:53:41.094769 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:53:41.095147 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:53:41.095809 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:53:41.095930 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 23:53:41.096033 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 23:53:41.096102 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:53:41.105592 27999 raft_consensus.cc:1261] T 13d6d62076a44fb280ffa7ea65572930 P 293be72579c14d4289f6c56e6867181f [term 1 FOLLOWER]: Refusing update from remote peer dc38ca701e594daa9c8c80eef3d4f6db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:53:41.105978 28365 raft_consensus.cc:1261] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [term 1 FOLLOWER]: Refusing update from remote peer dc38ca701e594daa9c8c80eef3d4f6db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:53:41.106606 28487 consensus_queue.cc:1035] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [LEADER]: Connected to new peer: Peer: permanent_uuid: "293be72579c14d4289f6c56e6867181f" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:53:41.107126 28480 consensus_queue.cc:1035] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9038a15c04c4a8fb32d75162d4dd871" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:53:41.117587 28495 mvcc.cc:205] Tried to move back new op lower bound from 7315703484838686720 to 7315703484711084032. Current Snapshot: MvccSnapshot[applied={T|T < 7315703484838686720}]
I20260806 23:53:41.118175 28493 mvcc.cc:205] Tried to move back new op lower bound from 7315703484838686720 to 7315703484711084032. Current Snapshot: MvccSnapshot[applied={T|T < 7315703484838686720}]
I20260806 23:53:41.119138 28498 mvcc.cc:205] Tried to move back new op lower bound from 7315703484838686720 to 7315703484711084032. Current Snapshot: MvccSnapshot[applied={T|T < 7315703484838686720}]
I20260806 23:53:41.130084 27767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:53:41.130321 27767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:53:41.133255 27767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1547dde5eeee4936a438a7ea31daf546: Sending DeleteTablet for 3 replicas of tablet 13d6d62076a44fb280ffa7ea65572930
I20260806 23:53:41.134222 28343 tablet_service.cc:1514] Processing DeleteTablet for tablet 13d6d62076a44fb280ffa7ea65572930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:53:41 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:42990
I20260806 23:53:41.134228 28172 tablet_service.cc:1514] Processing DeleteTablet for tablet 13d6d62076a44fb280ffa7ea65572930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:53:41 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:36978
I20260806 23:53:41.134461 27955 tablet_service.cc:1514] Processing DeleteTablet for tablet 13d6d62076a44fb280ffa7ea65572930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:53:41 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:45576
I20260806 23:53:41.135648 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 27887
I20260806 23:53:41.149892 28509 tablet_replica.cc:330] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: stopping tablet replica
I20260806 23:53:41.150338 28509 raft_consensus.cc:2229] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:53:41.150882 28509 raft_consensus.cc:2258] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:53:41.152765 28509 ts_tablet_manager.cc:1905] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:53:41.155872 28509 ts_tablet_manager.cc:1918] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:53:41.155982 28509 log.cc:1196] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/13d6d62076a44fb280ffa7ea65572930
I20260806 23:53:41.156299 28509 ts_tablet_manager.cc:1939] T 13d6d62076a44fb280ffa7ea65572930 P e9038a15c04c4a8fb32d75162d4dd871: Deleting consensus metadata
I20260806 23:53:41.157807 27751 catalog_manager.cc:4841] TS e9038a15c04c4a8fb32d75162d4dd871 (127.10.225.131:42355): tablet 13d6d62076a44fb280ffa7ea65572930 (table test-table [id=adffb99dcd5b40bd90c61f1bf783e49b]) successfully deleted
I20260806 23:53:41.158366 28513 tablet_replica.cc:330] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: stopping tablet replica
I20260806 23:53:41.158761 28513 raft_consensus.cc:2229] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:53:41.159228 28513 raft_consensus.cc:2258] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:53:41.162618 28513 ts_tablet_manager.cc:1905] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:53:41.174446 28513 ts_tablet_manager.cc:1918] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:53:41.174579 28513 log.cc:1196] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/13d6d62076a44fb280ffa7ea65572930
I20260806 23:53:41.174906 28513 ts_tablet_manager.cc:1939] T 13d6d62076a44fb280ffa7ea65572930 P dc38ca701e594daa9c8c80eef3d4f6db: Deleting consensus metadata
I20260806 23:53:41.177102 27752 catalog_manager.cc:4841] TS dc38ca701e594daa9c8c80eef3d4f6db (127.10.225.130:37911): tablet 13d6d62076a44fb280ffa7ea65572930 (table test-table [id=adffb99dcd5b40bd90c61f1bf783e49b]) successfully deleted
W20260806 23:53:41.177618 27749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:34221 (error 108)
I20260806 23:53:41.179459 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:53:41.179121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:34221 (local address 127.0.0.1:45592)
0806 23:53:41.179303 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 23:53:41.179370 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:34221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260806 23:53:41.179625 27749 catalog_manager.cc:4584] TS 293be72579c14d4289f6c56e6867181f (127.10.225.129:34221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13d6d62076a44fb280ffa7ea65572930: Network error: Client connection negotiation failed: client connection to 127.10.225.129:34221: connect: Connection refused (error 111)
I20260806 23:53:41.180064 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 28082
I20260806 23:53:41.198330 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 28273
I20260806 23:53:41.216509 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 27721
I20260806 23:53:41.232498 11142 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 23:53:41.581153 11142 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 23:53:41.581354 11142 mini_ranger.cc:66] Stopping Ranger...
I20260806 23:53:41.934087 11142 mini_ranger.cc:68] Stopped Ranger
2026-08-06 23:53:41.934 UTC [27151] LOG: received smart shutdown request
2026-08-06 23:53:41.938 UTC [27151] LOG: background worker "logical replication launcher" (PID 27168) exited with exit code 1
2026-08-06 23:53:41.938 UTC [27157] LOG: shutting down
2026-08-06 23:53:42.093 UTC [27151] LOG: database system is shut down
2026-08-06T23:53:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52055 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:53:42 dist-test-slave-nj21 krb5kdc[28531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:53:42 dist-test-slave-nj21 krb5kdc[28531](info): set up 2 sockets
Aug 06 23:53:42 dist-test-slave-nj21 krb5kdc[28531](info): commencing operation
krb5kdc: starting...
W20260806 23:53:43.360126 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:53:43 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:53:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/rangeradmin_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/rangeradmin_127.10.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/rangerlookup_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/rangerlookup_127.10.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/HTTP_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/HTTP_127.10.225.148@KRBTEST.COM.keytab.
I20260806 23:53:43.546710 11142 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyweHpq/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/postgres -l logfile start
2026-08-06 23:53:45.959 UTC [28571] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 23:53:45.959 UTC [28571] LOG: listening on IPv4 address "127.10.225.148", port 54987
2026-08-06 23:53:45.962 UTC [28571] LOG: listening on Unix socket "/tmp/.s.PGSQL.54987"
2026-08-06 23:53:45.992 UTC [28575] LOG: database system was shut down at 2026-08-06 23:53:44 UTC
2026-08-06 23:53:45.995 UTC [28571] LOG: database system is ready to accept connections
I20260806 23:53:47.047156 11142 mini_postgres.cc:96] Postgres bound to 54987
2026-08-06 23:53:47.055 UTC [28587] FATAL: database "slave" does not exist
127.10.225.148:54987 - accepting connections
I20260806 23:53:47.056628 11142 mini_ranger.cc:161] Starting Ranger...
I20260806 23:53:47.073266 11142 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 23:53:47.515064 11142 mini_ranger.cc:87] Created ranger Postgres database
I20260806 23:53:47.515139 11142 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-admin
2026-08-06 23:53:47,812 [I] DB FLAVOR :POSTGRES
2026-08-06 23:53:47,813 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 23:53:47,813 [I] Checking connection..
2026-08-06 23:53:47,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 23:53:48,083 [I] Checking connection passed.
2026-08-06 23:53:48,083 [I] --------- Verifying version history table ---------
2026-08-06 23:53:48,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:53:48,359 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 23:53:48,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:53:48,645 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 23:53:48,645 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 23:53:48,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 23:53:48.924 UTC [28670] WARNING: there is no transaction in progress
2026-08-06 23:53:48,933 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 23:53:48,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 23:53:49,215 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 23:53:49,216 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 23:53:49,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 23:53:49,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 23:53:49,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nj21.c.gcp-upstream.internal','N') ;"
2026-08-06 23:53:50,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 23:53:50,278 [I] Table x_portal_user does not exist in database ranger
2026-08-06 23:53:50,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 23:53:50,554 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 23:53:50,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 23:53:50,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 23:53:51,087 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 23:53:51,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 23:53:52.070 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.088 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.104 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.120 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.137 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.156 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.212 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.219 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.233 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.243 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.254 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.268 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.278 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.285 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.291 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52.299 UTC [28826] WARNING: there is no transaction in progress
2026-08-06 23:53:52,952 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 23:53:52,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 23:53:53,203 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 23:53:53,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 23:53:53,474 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 23:53:53,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 23:53:53,755 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 23:53:53,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 23:53:54,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 23:53:54,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nj21.c.gcp-upstream.internal';"
2026-08-06 23:53:54,550 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 23:53:54,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 23:53:54,806 [I] DB_PATCHES have already been applied
I20260806 23:53:54.812721 11142 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyweHpq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 23:53:54.812790 11142 mini_ranger.cc:193] Using host: 127.10.225.148
I20260806 23:53:54.817343 11142 mini_ranger.cc:227] Ranger admin URL: http://127.10.225.148:54235
Aug 06, 2026 11:53:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:53:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:53:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 11:53:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 11:53:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 11:53:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 11:53:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 23:53:56.057979 11142 mini_ranger.cc:160] Time spent starting Ranger: real 9.001s user 0.003s sys 0.009s
Aug 06, 2026 11:54:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 11:54:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 11:54:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 11:54:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 11:54:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 23:54:20.016958 11142 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.225.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.225.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/rangerkms_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.225.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/rangerkms_127.10.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.225.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.225.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.225.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/HTTP_127.10.225.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.225.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/HTTP_127.10.225.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 23:54:20 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060460, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 23:54:20.103783 11142 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 23:54:20.114966 11142 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 23:54:20.895085 11142 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 23:54:20.895174 11142 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-kms
I20260806 23:54:20.895192 11142 mini_ranger_kms.cc:227] Using postgres at 127.10.225.148:54987
2026-08-06 23:54:21,076 [I] DB FLAVOR :POSTGRES
2026-08-06 23:54:21,076 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 23:54:21,077 [I] Checking connection
2026-08-06 23:54:21,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 23:54:21,346 [I] connection success
2026-08-06 23:54:21,347 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 23:54:21,347 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 23:54:21,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 23:54:21,611 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 23:54:21,611 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 23:54:21,611 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 23:54:21,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.225.148:54987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 23:54:21,932 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 23:54:22.467861 11142 mini_ranger_kms.cc:310] Created kms service
I20260806 23:54:22.701781 11142 mini_ranger_kms.cc:326] Created kudu user
I20260806 23:54:22.752650 11142 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 23:54:22 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060460, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:23.215405 11142 mini_ranger_kms.cc:384] Added ranger policy
I20260806 23:54:23.215713 11142 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyweHpq/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyweHpq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 23:54:23.215746 11142 mini_ranger_kms.cc:240] Using host: 127.10.225.148
I20260806 23:54:23.220297 11142 mini_ranger_kms.cc:276] Ranger KMS PID: 29082
I20260806 23:54:23.220382 11142 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.225.148:53545
Aug 06, 2026 11:54:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 11:54:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 11:54:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 11:54:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 11:54:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 11:54:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 23:54:25 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060465, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.225.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:25.708429 11142 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.605s user 0.017s sys 0.149s
I20260806 23:54:25.708566 11142 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 23:54:25.708631 11142 mini_ranger_kms.cc:401] 127.10.225.148:53545/kms/v1/keys
Aug 06 23:54:26 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060465, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.225.148@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
Aug 06 23:54:26 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060465, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.225.148@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
Aug 06 23:54:28 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:54:28.959934 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35293
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35293
--ranger_config_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:53545/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:29.083187 29159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:29.083613 29159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:29.083689 29159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:29.089808 29159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:54:29.089895 29159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:29.089916 29159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:29.089939 29159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:54:29.089956 29159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:54:29.096385 29159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.225.148:53545/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35293
--ranger_config_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:29.097535 29159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:29.098579 29159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:54:29.108240 29159 server_base.cc:1034] running on GCE node
W20260806 23:54:29.108363 29169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:29.108353 29170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:29.108690 29173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:29.109264 29159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:29.110988 29159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:29.112185 29159 hybrid_clock.cc:648] HybridClock initialized: now 1786060469112168 us; error 49 us; skew 500 ppm
Aug 06 23:54:29 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060469, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:29.117941 29159 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:54:29.119037 29159 webserver.cc:460] Webserver started at http://127.10.225.190:42479/ using document root <none> and password file <none>
I20260806 23:54:29.119637 29159 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:29.119728 29159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:29.119973 29159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:54:29 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060469, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:29.181111 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "22bea8b83ba14e2984234927d1565e02"
format_stamp: "Formatted at 2026-08-06 23:54:29 on dist-test-slave-nj21"
server_key: "91a97287d074b959a52ae1aa4420fefb"
server_key_iv: "b32a650d3c1cccd44dc31b56cde246b2"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:29.181959 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "22bea8b83ba14e2984234927d1565e02"
format_stamp: "Formatted at 2026-08-06 23:54:29 on dist-test-slave-nj21"
server_key: "91a97287d074b959a52ae1aa4420fefb"
server_key_iv: "b32a650d3c1cccd44dc31b56cde246b2"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:29.187273 29159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 23:54:29.207165 29182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:29.208807 29159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:29.208954 29159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "22bea8b83ba14e2984234927d1565e02"
format_stamp: "Formatted at 2026-08-06 23:54:29 on dist-test-slave-nj21"
server_key: "91a97287d074b959a52ae1aa4420fefb"
server_key_iv: "b32a650d3c1cccd44dc31b56cde246b2"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:29.209084 29159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:29.227732 29159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:29.231002 29159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:29.231225 29159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:29.239698 29159 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:35293
I20260806 23:54:29.239717 29248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:35293 every 8 connection(s)
I20260806 23:54:29.240767 29159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:54:29.243652 29249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:29.249611 29249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Bootstrap starting.
I20260806 23:54:29.250022 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 29159
I20260806 23:54:29.250136 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:54:29.250396 11142 external_mini_cluster.cc:1428] Setting key bb8358adfa5e93738f00cb806e0ad4d1
I20260806 23:54:29.252442 29249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:29.253407 29249 log.cc:824] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:29.255697 29249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: No bootstrap required, opened a new log
Aug 06 23:54:29 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:29.259912 29249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } }
I20260806 23:54:29.260196 29249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:29.260264 29249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22bea8b83ba14e2984234927d1565e02, State: Initialized, Role: FOLLOWER
I20260806 23:54:29.260728 29249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } }
I20260806 23:54:29.260901 29249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:54:29.260974 29249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:54:29.261078 29249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:29.262518 29249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } }
I20260806 23:54:29.262920 29249 leader_election.cc:302] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22bea8b83ba14e2984234927d1565e02; no voters:
I20260806 23:54:29.263289 29249 leader_election.cc:288] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:54:29.263700 29255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:54:29.264032 29255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 1 LEADER]: Becoming Leader. State: Replica: 22bea8b83ba14e2984234927d1565e02, State: Running, Role: LEADER
I20260806 23:54:29.264412 29255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } }
I20260806 23:54:29.264716 29249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:54:29.266559 29261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22bea8b83ba14e2984234927d1565e02. Latest consensus state: current_term: 1 leader_uuid: "22bea8b83ba14e2984234927d1565e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } } }
I20260806 23:54:29.266593 29256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22bea8b83ba14e2984234927d1565e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } } }
I20260806 23:54:29.266789 29261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:29.266808 29256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:29.266654 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:29.251854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.10.225.190:35293)
0806 23:54:29.252347 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:29.252368 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:29.252413 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:54:29.253307 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:29.253331 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:29.253747 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:29.254074 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:29.254890 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:29.256379 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:29.257302 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:29.257986 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:29.258145 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:29.260743 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:29.260750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:29.260767 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:29.260801 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:29.262728 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:29.263220 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:29.263226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:29.263232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:29.263381 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:29.263740 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:29.263743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:29.263745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:29.264243 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:29.264459 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:29.265094 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 23:54:29.265631 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":132,"threads_started":1}
I20260806 23:54:29.267076 29249 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 23:54:29.268326 29281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:54:29.274870 29281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:54:29.282423 29281 catalog_manager.cc:1347] Generated new cluster ID: 481f293114004a9bb5899276a83dfb5c
I20260806 23:54:29.282500 29281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:54:29.292505 29281 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:54:29.293471 29281 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:54:29.298117 29281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Generated new TSK 0
I20260806 23:54:29.298770 29281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:54:30.098534 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:53545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35293
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:30.266830 29320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:30.267146 29320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:30.267282 29320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:30.274125 29320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:30.274202 29320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:30.274283 29320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:54:30.282045 29320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.225.148:53545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:30.283916 29320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:30.286046 29320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:30.298566 29333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:30.298806 29336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:30.298616 29332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:30.299808 29320 server_base.cc:1034] running on GCE node
I20260806 23:54:30.300318 29320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:30.301157 29320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:30.302358 29320 hybrid_clock.cc:648] HybridClock initialized: now 1786060470302333 us; error 41 us; skew 500 ppm
Aug 06 23:54:30 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060470, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:30.306138 29320 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:54:30.307178 29320 webserver.cc:460] Webserver started at http://127.10.225.129:45361/ using document root <none> and password file <none>
I20260806 23:54:30.307780 29320 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:30.307843 29320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:30.308068 29320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:54:30 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060470, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:30.329710 29320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "202c9828568d450f894b2810f2d6ee79"
format_stamp: "Formatted at 2026-08-06 23:54:30 on dist-test-slave-nj21"
server_key: "32fdc99ce2cb00acca89bb77f1ef4e65"
server_key_iv: "100419e10945ef3fc5d20d1fb675e830"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:30.330565 29320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "202c9828568d450f894b2810f2d6ee79"
format_stamp: "Formatted at 2026-08-06 23:54:30 on dist-test-slave-nj21"
server_key: "32fdc99ce2cb00acca89bb77f1ef4e65"
server_key_iv: "100419e10945ef3fc5d20d1fb675e830"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:30.348240 29320 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.001s sys 0.004s
I20260806 23:54:30.372776 29345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:30.374543 29320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 23:54:30.374682 29320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "202c9828568d450f894b2810f2d6ee79"
format_stamp: "Formatted at 2026-08-06 23:54:30 on dist-test-slave-nj21"
server_key: "32fdc99ce2cb00acca89bb77f1ef4e65"
server_key_iv: "100419e10945ef3fc5d20d1fb675e830"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:30.374814 29320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:30.422822 29320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:30.426864 29320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:30.427137 29320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:30.427965 29320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:30.429159 29320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:30.429270 29320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:30.429545 29320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:30.429651 29320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:30.444697 29320 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:45113
I20260806 23:54:30.446007 29320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:54:30.448041 29506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:45113 every 8 connection(s)
I20260806 23:54:30.451363 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 29320
I20260806 23:54:30.451484 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:54:30.451812 11142 external_mini_cluster.cc:1428] Setting key 18d7e3b6c8e12a86e0a3915ddbc5644f
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
Aug 06 23:54:30 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060470, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
Aug 06 23:54:30 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060470, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:30.480458 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:30.450102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:38999 (local address 127.10.225.190:35293)
0806 23:54:30.457404 (+ 7302us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:30.457409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:30.458834 (+ 1425us) server_negotiation.cc:407] Connection header received
0806 23:54:30.458886 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:30.458889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:30.458944 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:30.459038 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:30.460655 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:30.461521 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:30.463280 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:30.463930 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:30.464042 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:30.474771 (+ 10729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:30.474777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:30.474780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:30.474818 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:30.476858 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:30.478271 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:30.478276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:30.478278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:30.478382 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:30.478817 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:30.478820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:30.478822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:30.479097 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:30.479184 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:30.480029 (+ 845us) server_negotiation.cc:299] Negotiation successful
0806 23:54:30.480254 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7221,"thread_start_us":104,"threads_started":1}
I20260806 23:54:30.482709 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:30.450586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35293 (local address 127.10.225.129:38999)
0806 23:54:30.457291 (+ 6705us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:30.457369 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:30.458492 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:30.459300 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:30.459311 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:30.459708 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:30.460455 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:30.460479 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:30.461688 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:30.461692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:30.463128 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:30.463141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:30.464114 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:30.464119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:30.464306 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:30.466639 (+ 2333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:30.466658 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:30.474456 (+ 7798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:30.477645 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:30.477653 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:30.477682 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:30.478134 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:30.478500 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:30.478503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:30.478505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:30.478704 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:30.479262 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:30.479270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:30.479667 (+ 397us) client_negotiation.cc:769] Sending connection context
0806 23:54:30.480996 (+ 1329us) client_negotiation.cc:240] Negotiation successful
0806 23:54:30.481489 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6412,"thread_start_us":148,"threads_started":1}
I20260806 23:54:30.484659 29508 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35293
I20260806 23:54:30.484941 29508 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:30.485580 29508 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
I20260806 23:54:30.487581 29205 ts_manager.cc:194] Registered new tserver with Master: 202c9828568d450f894b2810f2d6ee79 (127.10.225.129:45113)
I20260806 23:54:30.489421 29205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:38999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:54:30.496748 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:53545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35293
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:30.673897 29521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:30.674340 29521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:30.674479 29521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:30.682111 29521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:30.682215 29521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:30.682403 29521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:54:30.691520 29521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.225.148:53545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:30.693292 29521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:30.694747 29521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:30.708220 29539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:30.708231 29528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:30.708674 29529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:30.709662 29521 server_base.cc:1034] running on GCE node
I20260806 23:54:30.710338 29521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:30.711194 29521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:30.712486 29521 hybrid_clock.cc:648] HybridClock initialized: now 1786060470712399 us; error 103 us; skew 500 ppm
Aug 06 23:54:30 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060470, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:30.716465 29521 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:54:30.717724 29521 webserver.cc:460] Webserver started at http://127.10.225.130:41835/ using document root <none> and password file <none>
I20260806 23:54:30.718588 29521 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:30.718700 29521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:30.718967 29521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:54:30 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060470, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:30.742839 29521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "4fc7572f75ea49238f54580df9fc8ae7"
format_stamp: "Formatted at 2026-08-06 23:54:30 on dist-test-slave-nj21"
server_key: "99752ba814d50905994eabcf06ff1cde"
server_key_iv: "59cacdefecbcfdb11951b7ad2cbe9d4a"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:30.743629 29521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "4fc7572f75ea49238f54580df9fc8ae7"
format_stamp: "Formatted at 2026-08-06 23:54:30 on dist-test-slave-nj21"
server_key: "99752ba814d50905994eabcf06ff1cde"
server_key_iv: "59cacdefecbcfdb11951b7ad2cbe9d4a"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:30.749572 29521 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 23:54:30.787321 29547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:30.789232 29521 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260806 23:54:30.789395 29521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "4fc7572f75ea49238f54580df9fc8ae7"
format_stamp: "Formatted at 2026-08-06 23:54:30 on dist-test-slave-nj21"
server_key: "99752ba814d50905994eabcf06ff1cde"
server_key_iv: "59cacdefecbcfdb11951b7ad2cbe9d4a"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:30.789574 29521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:30.819017 29521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:30.822240 29521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:30.822449 29521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:30.823077 29521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:30.824139 29521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:30.824198 29521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:30.824306 29521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:30.824339 29521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:30.837788 29521 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:41265
I20260806 23:54:30.839250 29521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:54:30.840423 29709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:41265 every 8 connection(s)
I20260806 23:54:30.847343 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 29521
I20260806 23:54:30.847478 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:54:30.847841 11142 external_mini_cluster.cc:1428] Setting key b35f01823eff232fb36481e52cd536f4
Aug 06 23:54:30 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060470, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:30.862515 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:30.844515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:51473 (local address 127.10.225.190:35293)
0806 23:54:30.844681 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:30.844686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:30.845495 (+ 809us) server_negotiation.cc:407] Connection header received
0806 23:54:30.846706 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:30.846709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:30.846770 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:30.846878 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:30.848471 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:30.849299 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:30.853696 (+ 4397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:30.854085 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:30.854208 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:30.857461 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:30.857466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:30.857469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:30.857503 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:30.859103 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:30.859661 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:30.859664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:30.859666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:30.859788 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:30.860225 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:30.860229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:30.860230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:30.860796 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:30.860971 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:30.862056 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0806 23:54:30.862288 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:54:30.865267 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:30.843137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35293 (local address 127.10.225.130:51473)
0806 23:54:30.845286 (+ 2149us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:30.845345 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:30.846479 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:30.847098 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:30.847109 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:30.847508 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:30.848278 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:30.848302 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:30.852025 (+ 3723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:30.852030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:30.853543 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:30.853559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:30.854265 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:30.854271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:30.854510 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:30.855306 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:30.855330 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:30.857219 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:30.859247 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:30.859254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:30.859273 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:30.859543 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:30.859898 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:30.859901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:30.859903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:30.860104 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:30.861118 (+ 1014us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:30.861129 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:30.861678 (+ 549us) client_negotiation.cc:769] Sending connection context
0806 23:54:30.863155 (+ 1477us) client_negotiation.cc:240] Negotiation successful
0806 23:54:30.863647 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1888,"thread_start_us":172,"threads_started":1}
I20260806 23:54:30.867569 29712 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35293
I20260806 23:54:30.867903 29712 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:30.868647 29712 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
I20260806 23:54:30.870234 29205 ts_manager.cc:194] Registered new tserver with Master: 4fc7572f75ea49238f54580df9fc8ae7 (127.10.225.130:41265)
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:54:30.871060 29205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:51473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:54:30.907478 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:53545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35293
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:54:31 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060470, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
W20260806 23:54:31.089144 29718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:31.089819 29718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:31.089955 29718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:31.096345 29718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:31.096431 29718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:31.096519 29718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:54:31.105433 29718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.225.148:53545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:35293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:31.107378 29718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:31.108788 29718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:31.130829 29723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:31.131508 29718 server_base.cc:1034] running on GCE node
W20260806 23:54:31.135854 29726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:31.136143 29724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:31.136785 29718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:31.137681 29718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:31.138862 29718 hybrid_clock.cc:648] HybridClock initialized: now 1786060471138791 us; error 89 us; skew 500 ppm
Aug 06 23:54:31 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:31.143085 29718 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:54:31.144367 29718 webserver.cc:460] Webserver started at http://127.10.225.131:38649/ using document root <none> and password file <none>
I20260806 23:54:31.145131 29718 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:31.145435 29718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:31.145768 29718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 23:54:31 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:31.168776 29718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "6eab6024af434ee7b67c6b66d30cd108"
format_stamp: "Formatted at 2026-08-06 23:54:31 on dist-test-slave-nj21"
server_key: "db4e2fbe3556ccce0c3b95ab9ffde908"
server_key_iv: "1ea05f68a8ae29954563cf541cd9c85b"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:31.170151 29718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "6eab6024af434ee7b67c6b66d30cd108"
format_stamp: "Formatted at 2026-08-06 23:54:31 on dist-test-slave-nj21"
server_key: "db4e2fbe3556ccce0c3b95ab9ffde908"
server_key_iv: "1ea05f68a8ae29954563cf541cd9c85b"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:31.189627 29718 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.000s sys 0.004s
I20260806 23:54:31.220165 29734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:31.222039 29718 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260806 23:54:31.222193 29718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "6eab6024af434ee7b67c6b66d30cd108"
format_stamp: "Formatted at 2026-08-06 23:54:31 on dist-test-slave-nj21"
server_key: "db4e2fbe3556ccce0c3b95ab9ffde908"
server_key_iv: "1ea05f68a8ae29954563cf541cd9c85b"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:31.222342 29718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:31.275326 29718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:31.278518 29718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:31.278740 29718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:31.279440 29718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:31.280519 29718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:31.280580 29718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:31.280683 29718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:31.280730 29718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:31.293699 29718 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:38773
I20260806 23:54:31.294757 29718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:54:31.295817 29895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:38773 every 8 connection(s)
I20260806 23:54:31.304579 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 29718
I20260806 23:54:31.304728 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
Aug 06 23:54:31 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:31.305079 11142 external_mini_cluster.cc:1428] Setting key f16405941f7ce6e42611bf81b5d7c322
I20260806 23:54:31.310038 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:31.296962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:57833 (local address 127.10.225.190:35293)
0806 23:54:31.297113 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:31.297119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:31.298111 (+ 992us) server_negotiation.cc:407] Connection header received
0806 23:54:31.299200 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:31.299204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:31.299267 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:31.299353 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:31.300620 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:31.301171 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:31.302333 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:31.302563 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:31.302658 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:31.305494 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:31.305498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:31.305501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:31.305532 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:31.307358 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:31.308240 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:31.308244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:31.308247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:31.308338 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:31.308707 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:31.308711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:31.308713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:31.309008 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:31.309112 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:31.309699 (+ 587us) server_negotiation.cc:299] Negotiation successful
0806 23:54:31.309846 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 23:54:31.310992 29900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:31.297392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35293 (local address 127.10.225.131:57833)
0806 23:54:31.297945 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:31.298001 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:31.298980 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:31.299547 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:31.299554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:31.299901 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:31.300416 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:31.300431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:31.301308 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:31.301311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:31.302193 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:31.302204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:31.302731 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:31.302736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:31.302923 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:31.303519 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:31.303538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:31.305273 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:31.307666 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:31.307700 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:31.307730 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:31.308086 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:31.308434 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:31.308438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:31.308440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:31.308597 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:31.309126 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:31.309134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:31.309415 (+ 281us) client_negotiation.cc:769] Sending connection context
0806 23:54:31.309617 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 23:54:31.309984 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":157,"threads_started":1}
I20260806 23:54:31.312155 29896 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35293
I20260806 23:54:31.312420 29896 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:31.313009 29896 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
I20260806 23:54:31.314112 29205 ts_manager.cc:194] Registered new tserver with Master: 6eab6024af434ee7b67c6b66d30cd108 (127.10.225.131:38773)
I20260806 23:54:31.314733 29205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:57833
I20260806 23:54:31.319546 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:54:31.320354 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 29320
I20260806 23:54:31.354779 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 29521
I20260806 23:54:31.371037 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 29718
I20260806 23:54:31.396530 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 29159
I20260806 23:54:31.413975 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35293
--webserver_interface=127.10.225.190
--webserver_port=42479
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35293
--ranger_config_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:53545/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.225.148:53545/kms with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:31.535914 29901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:31.536193 29901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:31.536234 29901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:31.540406 29901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:54:31.540503 29901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:31.540524 29901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:31.540539 29901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:54:31.540555 29901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:54:31.546864 29901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.225.148:53545/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35293
--ranger_config_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=42479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:31.548056 29901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:31.549090 29901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:31.557765 29906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:31.557896 29901 server_base.cc:1034] running on GCE node
W20260806 23:54:31.557771 29910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:31.557775 29907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:31.558553 29901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:31.559648 29901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:31.560883 29901 hybrid_clock.cc:648] HybridClock initialized: now 1786060471560837 us; error 57 us; skew 500 ppm
Aug 06 23:54:31 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:31.564417 29901 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:54:31.565869 29901 webserver.cc:460] Webserver started at http://127.10.225.190:42479/ using document root <none> and password file <none>
I20260806 23:54:31.566627 29901 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:31.566735 29901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 23:54:31 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:31.614610 29901 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:54:31.617435 29919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:31.619064 29901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:54:31.619213 29901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "22bea8b83ba14e2984234927d1565e02"
format_stamp: "Formatted at 2026-08-06 23:54:29 on dist-test-slave-nj21"
server_key: "91a97287d074b959a52ae1aa4420fefb"
server_key_iv: "b32a650d3c1cccd44dc31b56cde246b2"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:31.619874 29901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:31.640913 29901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:31.644670 29901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:31.644901 29901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:31.654040 29901 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:35293
I20260806 23:54:31.654104 29996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:35293 every 8 connection(s)
I20260806 23:54:31.655172 29901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:54:31.660586 29997 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:54:31.662822 29997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Bootstrap starting.
I20260806 23:54:31.662878 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 29901
I20260806 23:54:31.663353 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:53545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:45113
--local_ip_for_outbound_sockets=127.10.225.129
--tserver_master_addrs=127.10.225.190:35293
--webserver_port=45361
--webserver_interface=127.10.225.129
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.225.148:53545/kms with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:54:31.683372 29997 log.cc:824] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:31.690357 29997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:54:31.690749 29997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Bootstrap complete.
I20260806 23:54:31.694835 29997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } }
I20260806 23:54:31.695242 29997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22bea8b83ba14e2984234927d1565e02, State: Initialized, Role: FOLLOWER
I20260806 23:54:31.695792 29997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } }
I20260806 23:54:31.695926 29997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:54:31.695994 29997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:54:31.696089 29997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:54:31.698017 29997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } }
I20260806 23:54:31.698350 29997 leader_election.cc:302] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22bea8b83ba14e2984234927d1565e02; no voters:
I20260806 23:54:31.698659 29997 leader_election.cc:288] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 23:54:31.698784 30005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 23:54:31.698994 30005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [term 2 LEADER]: Becoming Leader. State: Replica: 22bea8b83ba14e2984234927d1565e02, State: Running, Role: LEADER
I20260806 23:54:31.699362 30005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } }
I20260806 23:54:31.700013 29997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:54:31.700824 30006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "22bea8b83ba14e2984234927d1565e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } } }
I20260806 23:54:31.700960 30006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:31.701223 30007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22bea8b83ba14e2984234927d1565e02. Latest consensus state: current_term: 2 leader_uuid: "22bea8b83ba14e2984234927d1565e02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22bea8b83ba14e2984234927d1565e02" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35293 } } }
I20260806 23:54:31.701377 30007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02 [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:31.702138 30016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:54:31.702421 29997 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 23:54:31.709767 30016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:54:31.710820 30016 catalog_manager.cc:1259] Loaded cluster ID: 481f293114004a9bb5899276a83dfb5c
I20260806 23:54:31.710899 30016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:54:31.713841 30016 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:54:31.714660 30016 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Loaded TSK: 0
I20260806 23:54:31.716102 30016 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 23:54:31.795523 30001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:31.795825 30001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:31.795888 30001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:31.800745 30001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:31.800915 30001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:31.801059 30001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:54:31.808167 30001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.225.148:53545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:45113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=45361
--tserver_master_addrs=127.10.225.190:35293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:31.809362 30001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:31.810415 30001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:31.824554 30054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:31.824687 30052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:31.824918 30051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:31.825824 30001 server_base.cc:1034] running on GCE node
I20260806 23:54:31.826778 30001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:31.827858 30001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:31.829114 30001 hybrid_clock.cc:648] HybridClock initialized: now 1786060471829083 us; error 45 us; skew 500 ppm
Aug 06 23:54:31 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:31.832494 30001 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:54:31.833544 30001 webserver.cc:460] Webserver started at http://127.10.225.129:45361/ using document root <none> and password file <none>
I20260806 23:54:31.834276 30001 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:31.834367 30001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 23:54:31 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:31.862334 30001 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:54:31.865908 30074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:31.867918 30001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:54:31.868106 30001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "202c9828568d450f894b2810f2d6ee79"
format_stamp: "Formatted at 2026-08-06 23:54:30 on dist-test-slave-nj21"
server_key: "32fdc99ce2cb00acca89bb77f1ef4e65"
server_key_iv: "100419e10945ef3fc5d20d1fb675e830"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:31.868901 30001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:31.886471 30001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:31.890381 30001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:31.890622 30001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:31.891376 30001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:31.892513 30001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:31.892570 30001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:31.892643 30001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:31.892669 30001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:31.904345 30001 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:45113
I20260806 23:54:31.905696 30001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:54:31.906567 30244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:45113 every 8 connection(s)
I20260806 23:54:31.910595 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 30001
I20260806 23:54:31.911083 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:53545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:41265
--local_ip_for_outbound_sockets=127.10.225.130
--tserver_master_addrs=127.10.225.190:35293
--webserver_port=41835
--webserver_interface=127.10.225.130
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.225.148:53545/kms with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:54:31 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060471, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:31.939062 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:31.911304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35293 (local address 127.10.225.129:57387)
0806 23:54:31.917420 (+ 6116us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:31.917498 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:31.918829 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:31.921339 (+ 2510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:31.921349 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:31.921732 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:31.922431 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:31.922454 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:31.924414 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:31.924418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:31.926003 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:31.926015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:31.927262 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:31.927348 (+ 86us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:31.927565 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:31.928427 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:31.928461 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:31.932299 (+ 3838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:31.934775 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:31.934781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:31.934798 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:31.935170 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:31.935606 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:31.935609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:31.935610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:31.935834 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:31.936703 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:31.936710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:31.937035 (+ 325us) client_negotiation.cc:769] Sending connection context
0806 23:54:31.937246 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 23:54:31.937630 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5820,"thread_start_us":142,"threads_started":1}
I20260806 23:54:31.941252 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:31.912785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:57387 (local address 127.10.225.190:35293)
0806 23:54:31.919505 (+ 6720us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:31.919529 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:31.919579 (+ 50us) server_negotiation.cc:407] Connection header received
0806 23:54:31.919938 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:31.919954 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:31.920413 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:31.920793 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:31.922660 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:31.924230 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:31.926161 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:31.926882 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:31.927064 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:31.932558 (+ 5494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:31.932565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:31.932580 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:31.932621 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:31.934617 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:31.935324 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:31.935330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:31.935336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:31.935491 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:31.935968 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:31.935972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:31.935973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:31.936505 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:31.936644 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:31.939898 (+ 3254us) server_negotiation.cc:299] Negotiation successful
0806 23:54:31.940229 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6535,"thread_start_us":113,"threads_started":1}
I20260806 23:54:31.949630 30245 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35293
I20260806 23:54:31.949956 30245 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:31.950919 30245 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
W20260806 23:54:31.954584 30245 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:35293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 23:54:31.954777 30245 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:31.955032 30245 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
I20260806 23:54:31.956183 30245 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:31.956393 30245 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
W20260806 23:54:31.957218 30245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:54:32.072697 30252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:32.072990 30252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:32.073040 30252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:32.079092 30252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:32.079200 30252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:32.079289 30252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:54:32.087497 30252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.225.148:53545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:41265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=41835
--tserver_master_addrs=127.10.225.190:35293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:32.089035 30252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:32.090750 30252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:32.101898 30257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:32.101930 30258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:32.101930 30260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:32.102636 30252 server_base.cc:1034] running on GCE node
I20260806 23:54:32.103171 30252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:32.104111 30252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:32.105381 30252 hybrid_clock.cc:648] HybridClock initialized: now 1786060472105343 us; error 54 us; skew 500 ppm
Aug 06 23:54:32 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:32.109697 30252 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:54:32.110952 30252 webserver.cc:460] Webserver started at http://127.10.225.130:41835/ using document root <none> and password file <none>
I20260806 23:54:32.111775 30252 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:32.111922 30252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 23:54:32 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:32.131843 30252 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:54:32.135809 30270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:32.137218 30252 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 23:54:32.137348 30252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "4fc7572f75ea49238f54580df9fc8ae7"
format_stamp: "Formatted at 2026-08-06 23:54:30 on dist-test-slave-nj21"
server_key: "99752ba814d50905994eabcf06ff1cde"
server_key_iv: "59cacdefecbcfdb11951b7ad2cbe9d4a"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:32.138062 30252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:32.152105 30252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:32.155229 30252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:32.155407 30252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:32.156107 30252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:32.157171 30252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:32.157272 30252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:32.157351 30252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:32.157389 30252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:32.168874 30252 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:41265
I20260806 23:54:32.170258 30252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:54:32.171489 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 30252
I20260806 23:54:32.172030 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.225.148:53545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:38773
--local_ip_for_outbound_sockets=127.10.225.131
--tserver_master_addrs=127.10.225.190:35293
--webserver_port=38649
--webserver_interface=127.10.225.131
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.225.148:53545/kms with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:54:32.172266 30439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:41265 every 8 connection(s)
Aug 06 23:54:32 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:32.188547 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:32.176038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:50599 (local address 127.10.225.190:35293)
0806 23:54:32.176171 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:32.176176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:32.176195 (+ 19us) server_negotiation.cc:407] Connection header received
0806 23:54:32.176902 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:32.176906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:32.176977 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:32.177030 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:32.178539 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.179068 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.180439 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.180745 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.180821 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:32.184130 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:32.184134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:32.184137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:32.184169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:32.185857 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:32.186535 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:32.186539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:32.186541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:32.186631 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:32.187060 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:32.187064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:32.187066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:32.187327 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:32.187416 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:32.188191 (+ 775us) server_negotiation.cc:299] Negotiation successful
0806 23:54:32.188363 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 23:54:32.190395 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:32.174963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35293 (local address 127.10.225.130:50599)
0806 23:54:32.175489 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:32.175543 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:32.176631 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:32.177448 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:32.177457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:32.177816 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:32.178383 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:32.178397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.179196 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.179199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:32.180298 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:32.180310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.181040 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.181044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:32.181232 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:32.182029 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:32.182050 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:32.183909 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:32.185997 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:32.186004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:32.186024 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:32.186407 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:32.186745 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:32.186748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:32.186750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:32.186940 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:32.187491 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:32.187498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:32.187881 (+ 383us) client_negotiation.cc:769] Sending connection context
0806 23:54:32.188852 (+ 971us) client_negotiation.cc:240] Negotiation successful
0806 23:54:32.189225 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":138,"threads_started":1}
I20260806 23:54:32.191928 30440 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35293
I20260806 23:54:32.192176 30440 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:32.192786 30440 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
W20260806 23:54:32.194581 30440 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:35293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 23:54:32.194700 30440 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:32.194868 30440 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
I20260806 23:54:32.195748 30440 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:32.195909 30440 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
W20260806 23:54:32.196684 30440 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:54:32.336213 30442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:32.336522 30442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:32.336587 30442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:32.342031 30442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:32.342345 30442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:32.342550 30442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:54:32.352548 30442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.225.148:53545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:38773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=38649
--tserver_master_addrs=127.10.225.190:35293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:32.353953 30442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:32.355176 30442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:32.366156 30451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:32.366539 30455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:32.366742 30442 server_base.cc:1034] running on GCE node
W20260806 23:54:32.366961 30452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:32.367501 30442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:32.368356 30442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:32.371721 30442 hybrid_clock.cc:648] HybridClock initialized: now 1786060472371668 us; error 42 us; skew 500 ppm
Aug 06 23:54:32 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:32.375058 30442 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:54:32.376202 30442 webserver.cc:460] Webserver started at http://127.10.225.131:38649/ using document root <none> and password file <none>
I20260806 23:54:32.376819 30442 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:32.376909 30442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 23:54:32 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:32.395785 30442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:32.399171 30464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:32.400938 30442 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:54:32.401074 30442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "6eab6024af434ee7b67c6b66d30cd108"
format_stamp: "Formatted at 2026-08-06 23:54:31 on dist-test-slave-nj21"
server_key: "db4e2fbe3556ccce0c3b95ab9ffde908"
server_key_iv: "1ea05f68a8ae29954563cf541cd9c85b"
server_key_version: "kuduclusterkey@0"
I20260806 23:54:32.401823 30442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:32.439265 30442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:32.443656 30442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:32.444037 30442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:32.444841 30442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:32.445945 30442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:32.446008 30442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:32.446064 30442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:32.446097 30442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:32.458720 30442 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:38773
I20260806 23:54:32.458763 30622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:38773 every 8 connection(s)
I20260806 23:54:32.460155 30442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:54:32.471463 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 30442
Aug 06 23:54:32 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:32.478511 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:32.463593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:48625 (local address 127.10.225.190:35293)
0806 23:54:32.463764 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:32.463769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:32.464903 (+ 1134us) server_negotiation.cc:407] Connection header received
0806 23:54:32.466090 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:32.466093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:32.466156 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:32.466258 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:32.467763 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.468596 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.470408 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.470785 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.470904 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:32.474026 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:32.474031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:32.474034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:32.474077 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:32.475888 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:32.476573 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:32.476577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:32.476579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:32.476680 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:32.477124 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:32.477128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:32.477129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:32.477349 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:32.477442 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:32.478191 (+ 749us) server_negotiation.cc:299] Negotiation successful
0806 23:54:32.478333 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:54:32.480325 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:32.464083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35293 (local address 127.10.225.131:48625)
0806 23:54:32.464700 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:32.464770 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:32.465814 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:32.466454 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:32.466463 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:32.466847 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:32.467529 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:32.467552 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.468778 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.468782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:32.470226 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:32.470238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.471111 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.471116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:32.471339 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:32.472084 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:32.472109 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:32.473751 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:32.476012 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:32.476020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:32.476041 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:32.476434 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:32.476784 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:32.476788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:32.476790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:32.477009 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:32.477467 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:32.477475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:32.477853 (+ 378us) client_negotiation.cc:769] Sending connection context
0806 23:54:32.478777 (+ 924us) client_negotiation.cc:240] Negotiation successful
0806 23:54:32.479169 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":216,"threads_started":1}
I20260806 23:54:32.481676 30623 heartbeater.cc:347] Connected to a master server at 127.10.225.190:35293
I20260806 23:54:32.481946 30623 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:32.482556 30623 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
W20260806 23:54:32.484372 30623 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:35293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 23:54:32.484510 30623 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:32.484719 30623 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
I20260806 23:54:32.486047 30623 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:32.486276 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:32.472677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.10.225.190:35293)
0806 23:54:32.475427 (+ 2750us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:32.475432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:32.475747 (+ 315us) server_negotiation.cc:407] Connection header received
0806 23:54:32.475964 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:32.475967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:32.476017 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:32.476067 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:32.476966 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.477642 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.478484 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:32.478774 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:32.478862 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:32.480379 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:32.480383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:32.480386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:32.480412 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:32.482132 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:32.484156 (+ 2024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:32.484160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:32.484162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:32.484263 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:32.484677 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:32.484680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:32.484682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:32.484976 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:32.485171 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:32.485881 (+ 710us) server_negotiation.cc:299] Negotiation successful
0806 23:54:32.486016 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2653,"thread_start_us":96,"threads_started":1}
I20260806 23:54:32.486392 30623 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
W20260806 23:54:32.488083 30623 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 23:54:32 dist-test-slave-nj21 krb5kdc[28531](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786060472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:32.957525 30245 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:32.957849 30245 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
I20260806 23:54:32.959008 29947 ts_manager.cc:194] Registered new tserver with Master: 202c9828568d450f894b2810f2d6ee79 (127.10.225.129:45113)
I20260806 23:54:32.960729 29947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:57387
I20260806 23:54:33.196900 30440 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:33.197244 30440 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
I20260806 23:54:33.197994 29947 ts_manager.cc:194] Registered new tserver with Master: 4fc7572f75ea49238f54580df9fc8ae7 (127.10.225.130:41265)
I20260806 23:54:33.198802 29947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:50599
Aug 06 23:54:33 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
I20260806 23:54:33.488418 30623 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:33.488796 30623 heartbeater.cc:511] Master 127.10.225.190:35293 requested a full tablet report, sending...
I20260806 23:54:33.489691 29947 ts_manager.cc:194] Registered new tserver with Master: 6eab6024af434ee7b67c6b66d30cd108 (127.10.225.131:38773)
I20260806 23:54:33.490489 29947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:48625
I20260806 23:54:33.496778 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:54:33.509809 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.498639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.10.225.190:35293)
0806 23:54:33.498928 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:33.498933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:33.499200 (+ 267us) server_negotiation.cc:407] Connection header received
0806 23:54:33.499348 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:33.499351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:33.499407 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:33.499498 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:33.500912 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.501879 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.503135 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.503501 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.503589 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.505377 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:33.505381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:33.505384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:33.505413 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:33.507296 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:33.507897 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:33.507901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:33.507903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:33.507994 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:33.508319 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:33.508323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:33.508325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:33.508558 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:33.508684 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:33.509365 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 23:54:33.509515 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":114,"threads_started":1}
I20260806 23:54:33.514735 29947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:54:33.519994 29947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:54:33.540650 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.533137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:38773 (local address 127.0.0.1:52898)
0806 23:54:33.534588 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:33.534600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:33.534697 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:33.535359 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:33.535365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:33.535390 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:33.535526 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.535532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.537396 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.537400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.538723 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.538734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.540068 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.540072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.540189 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.540228 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 23:54:33.540356 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 23:54:33.540490 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"thread_start_us":108,"threads_started":1}
I20260806 23:54:33.541369 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.533307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.10.225.131:38773)
0806 23:54:33.533890 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:33.533898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:33.534699 (+ 801us) server_negotiation.cc:407] Connection header received
0806 23:54:33.534827 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:33.534833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:33.535043 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:33.535181 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:33.535660 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.537254 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.538866 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.539735 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.539855 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.540002 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:33.541109 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0806 23:54:33.541220 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":104,"threads_started":1}
I20260806 23:54:33.543383 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.533471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:41265 (local address 127.0.0.1:46136)
0806 23:54:33.535100 (+ 1629us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:33.535113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:33.535268 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:33.535965 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:33.535967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:33.535987 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:33.536080 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.536086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.537991 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.537995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.541642 (+ 3647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.541654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.542978 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.542982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.543068 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.543093 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 23:54:33.543161 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 23:54:33.543232 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260806 23:54:33.543978 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.533632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.10.225.130:41265)
0806 23:54:33.534943 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:33.534949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:33.535195 (+ 246us) server_negotiation.cc:407] Connection header received
0806 23:54:33.535427 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:33.535433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:33.535612 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:33.535770 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:33.536211 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.537845 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.541786 (+ 3941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.542625 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.542744 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.542912 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:33.543661 (+ 749us) server_negotiation.cc:299] Negotiation successful
0806 23:54:33.543797 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":74,"threads_started":1}
I20260806 23:54:33.544816 30522 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4a1ba3ae444299865e5a03c2f71fc8 (DEFAULT_TABLE table=test-table [id=bb7d37c39908493a8effb344c72203af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:54:33.545969 30522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4a1ba3ae444299865e5a03c2f71fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:33.547379 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.532712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45113 (local address 127.0.0.1:33202)
0806 23:54:33.534149 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:33.534199 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:33.534329 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:33.540343 (+ 6014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:33.540347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:33.540379 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:33.540481 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.540487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.543725 (+ 3238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.543728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.544880 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.544892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.546965 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.546970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.547070 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.547097 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 23:54:33.547151 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 23:54:33.547224 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"thread_start_us":125,"threads_started":1}
I20260806 23:54:33.548115 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.533943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.10.225.129:45113)
0806 23:54:33.539838 (+ 5895us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:33.539845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:33.539872 (+ 27us) server_negotiation.cc:407] Connection header received
0806 23:54:33.539959 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:33.539965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:33.540183 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:33.540629 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:33.540778 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.543564 (+ 2786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.545191 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.546007 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.547664 (+ 1657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.547819 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:33.547873 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 23:54:33.547971 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5771,"thread_start_us":91,"threads_started":1}
I20260806 23:54:33.547819 30346 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4a1ba3ae444299865e5a03c2f71fc8 (DEFAULT_TABLE table=test-table [id=bb7d37c39908493a8effb344c72203af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:54:33.548863 30346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4a1ba3ae444299865e5a03c2f71fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:33.550995 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4a1ba3ae444299865e5a03c2f71fc8 (DEFAULT_TABLE table=test-table [id=bb7d37c39908493a8effb344c72203af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:54:33.552026 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4a1ba3ae444299865e5a03c2f71fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:33.560851 30671 tablet_bootstrap.cc:492] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: Bootstrap starting.
I20260806 23:54:33.563395 30671 tablet_bootstrap.cc:654] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:33.565105 30671 log.cc:824] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:33.566462 30672 tablet_bootstrap.cc:492] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79: Bootstrap starting.
I20260806 23:54:33.568130 30670 tablet_bootstrap.cc:492] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: Bootstrap starting.
I20260806 23:54:33.568969 30672 tablet_bootstrap.cc:654] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:33.569911 30672 log.cc:824] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:33.571048 30670 tablet_bootstrap.cc:654] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:33.572805 30672 tablet_bootstrap.cc:492] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79: No bootstrap required, opened a new log
I20260806 23:54:33.572808 30671 tablet_bootstrap.cc:492] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: No bootstrap required, opened a new log
I20260806 23:54:33.573025 30672 ts_tablet_manager.cc:1397] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 23:54:33.573025 30671 ts_tablet_manager.cc:1397] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20260806 23:54:33.577617 30672 raft_consensus.cc:348] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } }
I20260806 23:54:33.577945 30672 raft_consensus.cc:374] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:33.578013 30672 raft_consensus.cc:729] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202c9828568d450f894b2810f2d6ee79, State: Initialized, Role: FOLLOWER
I20260806 23:54:33.578166 30671 raft_consensus.cc:348] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } }
I20260806 23:54:33.578481 30671 raft_consensus.cc:374] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:33.578569 30671 raft_consensus.cc:729] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eab6024af434ee7b67c6b66d30cd108, State: Initialized, Role: FOLLOWER
I20260806 23:54:33.578538 30672 consensus_queue.cc:260] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } }
I20260806 23:54:33.579152 30671 consensus_queue.cc:260] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } }
I20260806 23:54:33.579784 30672 ts_tablet_manager.cc:1428] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 23:54:33.580191 30245 heartbeater.cc:503] Master 127.10.225.190:35293 was elected leader, sending a full tablet report...
I20260806 23:54:33.580220 30671 ts_tablet_manager.cc:1428] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 23:54:33.580606 30623 heartbeater.cc:503] Master 127.10.225.190:35293 was elected leader, sending a full tablet report...
I20260806 23:54:33.581216 30670 log.cc:824] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:33.589180 30670 tablet_bootstrap.cc:492] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: No bootstrap required, opened a new log
I20260806 23:54:33.589438 30670 ts_tablet_manager.cc:1397] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20260806 23:54:33.593389 30670 raft_consensus.cc:348] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } }
I20260806 23:54:33.593658 30670 raft_consensus.cc:374] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:33.593737 30670 raft_consensus.cc:729] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc7572f75ea49238f54580df9fc8ae7, State: Initialized, Role: FOLLOWER
I20260806 23:54:33.594206 30670 consensus_queue.cc:260] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } }
I20260806 23:54:33.594921 30440 heartbeater.cc:503] Master 127.10.225.190:35293 was elected leader, sending a full tablet report...
I20260806 23:54:33.595379 30670 ts_tablet_manager.cc:1428] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260806 23:54:33.658174 30247 tablet.cc:2369] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 23:54:33 dist-test-slave-nj21 krb5kdc[28531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060472, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for HTTP/127.10.225.148@KRBTEST.COM
W20260806 23:54:33.675299 30441 tablet.cc:2369] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:54:33.712249 30624 tablet.cc:2369] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:54:33.780284 30679 raft_consensus.cc:482] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:54:33.780555 30679 raft_consensus.cc:504] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } }
I20260806 23:54:33.781956 30679 leader_election.cc:288] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 202c9828568d450f894b2810f2d6ee79 (127.10.225.129:45113), 4fc7572f75ea49238f54580df9fc8ae7 (127.10.225.130:41265)
I20260806 23:54:33.792075 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.783376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:55991 (local address 127.10.225.129:45113)
0806 23:54:33.783543 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:33.783549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:33.787188 (+ 3639us) server_negotiation.cc:407] Connection header received
0806 23:54:33.787406 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:33.787409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:33.787461 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:33.787508 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:33.787994 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.788780 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.790579 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.791271 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.791378 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.791437 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:33.791825 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 23:54:33.791916 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:54:33.792826 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.782516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45113 (local address 127.10.225.131:55991)
0806 23:54:33.787066 (+ 4550us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:33.787086 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:33.787284 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:33.787655 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:33.787657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:33.787699 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:33.787849 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.787855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.788928 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.788942 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.790419 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.790433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.791540 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.791544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.791659 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.791708 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 23:54:33.792579 (+ 871us) client_negotiation.cc:240] Negotiation successful
0806 23:54:33.792669 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4452,"thread_start_us":116,"threads_started":1}
I20260806 23:54:33.793723 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4a1ba3ae444299865e5a03c2f71fc8" candidate_uuid: "6eab6024af434ee7b67c6b66d30cd108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202c9828568d450f894b2810f2d6ee79" is_pre_election: true
I20260806 23:54:33.794162 30162 raft_consensus.cc:2454] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eab6024af434ee7b67c6b66d30cd108 in term 0.
I20260806 23:54:33.795527 30468 leader_election.cc:302] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202c9828568d450f894b2810f2d6ee79, 6eab6024af434ee7b67c6b66d30cd108; no voters:
I20260806 23:54:33.795897 30679 raft_consensus.cc:2781] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:54:33.795990 30679 raft_consensus.cc:482] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:54:33.796031 30679 raft_consensus.cc:3037] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:33.797417 30679 raft_consensus.cc:504] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } }
I20260806 23:54:33.797832 30679 leader_election.cc:288] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [CANDIDATE]: Term 1 election: Requested vote from peers 202c9828568d450f894b2810f2d6ee79 (127.10.225.129:45113), 4fc7572f75ea49238f54580df9fc8ae7 (127.10.225.130:41265)
I20260806 23:54:33.798532 30162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4a1ba3ae444299865e5a03c2f71fc8" candidate_uuid: "6eab6024af434ee7b67c6b66d30cd108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202c9828568d450f894b2810f2d6ee79"
I20260806 23:54:33.798668 30162 raft_consensus.cc:3037] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:33.800087 30162 raft_consensus.cc:2454] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eab6024af434ee7b67c6b66d30cd108 in term 1.
I20260806 23:54:33.800570 30468 leader_election.cc:302] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202c9828568d450f894b2810f2d6ee79, 6eab6024af434ee7b67c6b66d30cd108; no voters:
I20260806 23:54:33.800840 30679 raft_consensus.cc:2781] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:54:33.801093 30679 raft_consensus.cc:686] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 1 LEADER]: Becoming Leader. State: Replica: 6eab6024af434ee7b67c6b66d30cd108, State: Running, Role: LEADER
I20260806 23:54:33.801529 30679 consensus_queue.cc:237] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } }
I20260806 23:54:33.805626 29947 catalog_manager.cc:5495] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6eab6024af434ee7b67c6b66d30cd108 (127.10.225.131). New cstate: current_term: 1 leader_uuid: "6eab6024af434ee7b67c6b66d30cd108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6eab6024af434ee7b67c6b66d30cd108" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38773 } health_report { overall_health: HEALTHY } } }
I20260806 23:54:33.810920 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.783535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:44077 (local address 127.10.225.130:41265)
0806 23:54:33.783651 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:33.783656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:33.793552 (+ 9896us) server_negotiation.cc:407] Connection header received
0806 23:54:33.793738 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:33.793742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:33.793793 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:33.793837 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:33.794366 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.795134 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.809499 (+ 14365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.810153 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.810253 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.810311 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:33.810682 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 23:54:33.810770 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 23:54:33.811530 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.782911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:41265 (local address 127.10.225.131:44077)
0806 23:54:33.793472 (+ 10561us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:33.793486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:33.793630 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:33.794065 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:33.794068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:33.794087 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:33.794164 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.794169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.807136 (+ 12967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.807140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.809297 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:33.809309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.810422 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.810425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:33.810536 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.810562 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 23:54:33.811297 (+ 735us) client_negotiation.cc:240] Negotiation successful
0806 23:54:33.811390 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10498,"thread_start_us":92,"threads_started":1}
I20260806 23:54:33.812382 30360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4a1ba3ae444299865e5a03c2f71fc8" candidate_uuid: "6eab6024af434ee7b67c6b66d30cd108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc7572f75ea49238f54580df9fc8ae7" is_pre_election: true
I20260806 23:54:33.812729 30360 raft_consensus.cc:2454] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eab6024af434ee7b67c6b66d30cd108 in term 0.
I20260806 23:54:33.813076 30365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4a1ba3ae444299865e5a03c2f71fc8" candidate_uuid: "6eab6024af434ee7b67c6b66d30cd108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc7572f75ea49238f54580df9fc8ae7"
I20260806 23:54:33.813189 30365 raft_consensus.cc:3037] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:33.814524 30365 raft_consensus.cc:2454] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eab6024af434ee7b67c6b66d30cd108 in term 1.
I20260806 23:54:33.850558 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:33.845207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.10.225.131:38773)
0806 23:54:33.845334 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:33.845339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:33.845354 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:54:33.845454 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:33.845457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:33.845527 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:33.845632 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:54:33.846014 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.846818 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.848257 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:33.848602 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:33.848699 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:33.849145 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:54:33.849766 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:54:33.849882 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:33.850333 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 23:54:33.850403 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:54:33.858809 30162 raft_consensus.cc:1261] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 202c9828568d450f894b2810f2d6ee79 [term 1 FOLLOWER]: Refusing update from remote peer 6eab6024af434ee7b67c6b66d30cd108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:54:33.859787 30365 raft_consensus.cc:1261] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [term 1 FOLLOWER]: Refusing update from remote peer 6eab6024af434ee7b67c6b66d30cd108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:54:33.860714 30688 consensus_queue.cc:1035] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc7572f75ea49238f54580df9fc8ae7" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:54:33.861783 30679 consensus_queue.cc:1035] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "202c9828568d450f894b2810f2d6ee79" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:54:33.878983 30694 mvcc.cc:205] Tried to move back new op lower bound from 7315703700914659328 to 7315703700695425024. Current Snapshot: MvccSnapshot[applied={T|T < 7315703700914659328}]
I20260806 23:54:33.882850 30698 mvcc.cc:205] Tried to move back new op lower bound from 7315703700914659328 to 7315703700695425024. Current Snapshot: MvccSnapshot[applied={T|T < 7315703700914659328}]
I20260806 23:54:33.897424 29947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:54:33.897686 29947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:54:33.901613 29947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22bea8b83ba14e2984234927d1565e02: Sending DeleteTablet for 3 replicas of tablet 1d4a1ba3ae444299865e5a03c2f71fc8
I20260806 23:54:33.902724 30522 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4a1ba3ae444299865e5a03c2f71fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:54:33 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:52898
I20260806 23:54:33.904080 30346 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4a1ba3ae444299865e5a03c2f71fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:54:33 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:46136
I20260806 23:54:33.905423 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 30001
I20260806 23:54:33.935954 30706 tablet_replica.cc:330] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: stopping tablet replica
I20260806 23:54:33.936319 30705 tablet_replica.cc:330] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: stopping tablet replica
I20260806 23:54:33.936919 30705 raft_consensus.cc:2229] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:54:33.937387 30706 raft_consensus.cc:2229] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:54:33.938446 30705 raft_consensus.cc:2258] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:54:33.938877 30706 raft_consensus.cc:2258] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:54:33.940850 30705 ts_tablet_manager.cc:1905] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:54:33.944024 30706 ts_tablet_manager.cc:1905] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:54:33.946898 30705 ts_tablet_manager.cc:1918] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:54:33.947049 30706 ts_tablet_manager.cc:1918] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:54:33.953154 30705 log.cc:1196] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/1d4a1ba3ae444299865e5a03c2f71fc8
I20260806 23:54:33.953395 30706 log.cc:1196] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/1d4a1ba3ae444299865e5a03c2f71fc8
I20260806 23:54:33.953583 30705 ts_tablet_manager.cc:1939] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 6eab6024af434ee7b67c6b66d30cd108: Deleting consensus metadata
I20260806 23:54:33.953887 30706 ts_tablet_manager.cc:1939] T 1d4a1ba3ae444299865e5a03c2f71fc8 P 4fc7572f75ea49238f54580df9fc8ae7: Deleting consensus metadata
I20260806 23:54:33.955113 29923 catalog_manager.cc:4841] TS 6eab6024af434ee7b67c6b66d30cd108 (127.10.225.131:38773): tablet 1d4a1ba3ae444299865e5a03c2f71fc8 (table test-table [id=bb7d37c39908493a8effb344c72203af]) successfully deleted
I20260806 23:54:33.955117 29925 catalog_manager.cc:4841] TS 4fc7572f75ea49238f54580df9fc8ae7 (127.10.225.130:41265): tablet 1d4a1ba3ae444299865e5a03c2f71fc8 (table test-table [id=bb7d37c39908493a8effb344c72203af]) successfully deleted
W20260806 23:54:33.958007 29925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:45113 (error 108)
I20260806 23:54:33.959869 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 30252
I20260806 23:54:33.993757 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 30442
I20260806 23:54:34.000725 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:34.000267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45113 (local address 127.0.0.1:33218)
0806 23:54:34.000532 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:34.000618 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:45113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260806 23:54:34.001681 29925 catalog_manager.cc:4584] TS 202c9828568d450f894b2810f2d6ee79 (127.10.225.129:45113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d4a1ba3ae444299865e5a03c2f71fc8: Network error: Client connection negotiation failed: client connection to 127.10.225.129:45113: connect: Connection refused (error 111)
I20260806 23:54:34.010456 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 29901
I20260806 23:54:34.023591 11142 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 23:54:34.374111 11142 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 23:54:34.374251 11142 mini_ranger.cc:66] Stopping Ranger...
I20260806 23:54:34.720191 11142 mini_ranger.cc:68] Stopped Ranger
2026-08-06 23:54:34.720 UTC [28571] LOG: received smart shutdown request
2026-08-06 23:54:34.725 UTC [28571] LOG: background worker "logical replication launcher" (PID 28585) exited with exit code 1
2026-08-06 23:54:34.725 UTC [28576] LOG: shutting down
2026-08-06 23:54:34.890 UTC [28571] LOG: database system is shut down
2026-08-06T23:54:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52792 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:54:35 dist-test-slave-nj21 krb5kdc[30723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:54:35 dist-test-slave-nj21 krb5kdc[30723](info): set up 2 sockets
Aug 06 23:54:35 dist-test-slave-nj21 krb5kdc[30723](info): commencing operation
krb5kdc: starting...
W20260806 23:54:36.145610 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:54:36.283166 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35587
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35587,127.10.225.189:45567,127.10.225.188:36065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:36.407043 30739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:36.407342 30739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:36.407402 30739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:36.411566 30739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:54:36.411646 30739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:36.411696 30739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:36.411720 30739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:54:36.411747 30739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:54:36.417424 30739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35587,127.10.225.189:45567,127.10.225.188:36065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:36.418581 30739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:36.419618 30739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:36.429226 30748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:36.429226 30751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:36.429278 30747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:36.429394 30739 server_base.cc:1034] running on GCE node
I20260806 23:54:36.430168 30739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:36.431484 30739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:36.432682 30739 hybrid_clock.cc:648] HybridClock initialized: now 1786060476432666 us; error 56 us; skew 500 ppm
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:36.436017 30739 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:54:36.437335 30739 webserver.cc:460] Webserver started at http://127.10.225.190:36085/ using document root <none> and password file <none>
I20260806 23:54:36.438112 30739 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:36.438198 30739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:36.438410 30739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:54:36.440618 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "3cee1ac085fd4934ace8328ef74247a8"
format_stamp: "Formatted at 2026-08-06 23:54:36 on dist-test-slave-nj21"
server_key: "bd168a84243f9e6c2447d27ac42ba591"
server_key_iv: "1b1026ad5b39286f9d60962c699cd627"
server_key_version: "encryptionkey@0"
I20260806 23:54:36.441141 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "3cee1ac085fd4934ace8328ef74247a8"
format_stamp: "Formatted at 2026-08-06 23:54:36 on dist-test-slave-nj21"
server_key: "bd168a84243f9e6c2447d27ac42ba591"
server_key_iv: "1b1026ad5b39286f9d60962c699cd627"
server_key_version: "encryptionkey@0"
I20260806 23:54:36.445969 30739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 23:54:36.448827 30761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:36.450255 30739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:54:36.450469 30739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "3cee1ac085fd4934ace8328ef74247a8"
format_stamp: "Formatted at 2026-08-06 23:54:36 on dist-test-slave-nj21"
server_key: "bd168a84243f9e6c2447d27ac42ba591"
server_key_iv: "1b1026ad5b39286f9d60962c699cd627"
server_key_version: "encryptionkey@0"
I20260806 23:54:36.450593 30739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:36.489645 30739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:36.493116 30739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:36.493319 30739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:36.500527 30739 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:35587
I20260806 23:54:36.500777 30836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:35587 every 8 connection(s)
I20260806 23:54:36.501893 30739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:54:36.503513 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 30739
I20260806 23:54:36.503643 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:54:36.503948 11142 external_mini_cluster.cc:1428] Setting key 973ca0ae0e15b4460e6df850ee018fbb
I20260806 23:54:36.505899 30839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:36.510340 30839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.189@KRBTEST.COM" created.
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:36.527555 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.512919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35587 (local address 127.0.0.1:48304)
0806 23:54:36.513583 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.513625 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:36.514493 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:36.515325 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:36.515332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:36.515813 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:36.516173 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.516194 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.517697 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.517700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.518434 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.518443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.519245 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.519248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.519372 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.520023 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:36.520041 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:36.521916 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:36.524134 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.524150 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.524170 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.524651 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.525008 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.525011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.525012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.525141 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.525825 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:36.525831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:36.525977 (+ 146us) client_negotiation.cc:769] Sending connection context
0806 23:54:36.526134 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 23:54:36.526507 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":51,"spinlock_wait_cycles":9088,"thread_start_us":160,"threads_started":1}
I20260806 23:54:36.527577 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.513791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.10.225.190:35587)
0806 23:54:36.514063 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.514072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.514115 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:54:36.514877 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.514882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.515114 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.515261 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.516355 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.517568 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.518562 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.519105 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.519297 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.522068 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:36.522076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:36.522091 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:36.522117 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:36.523899 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.524794 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.524799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.524801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.524921 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.525215 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.525217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.525218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.525663 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:36.525811 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:36.526244 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 23:54:36.526594 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab.
Entry for principal kudu/127.10.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab.
I20260806 23:54:36.530674 30839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:54:36.531052 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.189:45567
--webserver_interface=127.10.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab
--principal=kudu/127.10.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35587,127.10.225.189:45567,127.10.225.188:36065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:54:36.531462 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.531157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:45567 (local address 127.0.0.1:37752)
0806 23:54:36.531325 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.531395 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260806 23:54:36.531697 30763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
I20260806 23:54:36.532531 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.532339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:45567 (local address 127.0.0.1:37760)
0806 23:54:36.532435 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.532478 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 23:54:36.532653 30839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.189:45567: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
I20260806 23:54:36.581912 30839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } attempt: 1
I20260806 23:54:36.582886 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.582553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:45567 (local address 127.0.0.1:37776)
0806 23:54:36.582732 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.582801 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 23:54:36.583658 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.583454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:45567 (local address 127.0.0.1:37778)
0806 23:54:36.583557 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.583604 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 23:54:36.583812 30839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.189:45567: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
I20260806 23:54:36.651957 30839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } attempt: 2
I20260806 23:54:36.652865 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.652568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:45567 (local address 127.0.0.1:37792)
0806 23:54:36.652726 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.652788 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 23:54:36.653555 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.653357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:45567 (local address 127.0.0.1:37802)
0806 23:54:36.653459 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.653504 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 23:54:36.653657 30839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.189:45567: Network error: Client connection negotiation failed: client connection to 127.10.225.189:45567: connect: Connection refused (error 111)
W20260806 23:54:36.653889 30849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:36.654136 30849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:36.654182 30849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:36.658331 30849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:54:36.658386 30849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:36.658404 30849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:36.658420 30849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:54:36.658434 30849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:54:36.664924 30849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35587,127.10.225.189:45567,127.10.225.188:36065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab
--principal=kudu/127.10.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.189:45567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:36.666365 30849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:36.667415 30849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:36.677119 30860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:36.677119 30857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:36.677119 30858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:36.677412 30849 server_base.cc:1034] running on GCE node
I20260806 23:54:36.678226 30849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:36.679394 30849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:36.680569 30849 hybrid_clock.cc:648] HybridClock initialized: now 1786060476680535 us; error 56 us; skew 500 ppm
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:36.683902 30849 init.cc:377] Logged in from keytab as kudu/127.10.225.189@KRBTEST.COM (short username kudu)
I20260806 23:54:36.684861 30849 webserver.cc:460] Webserver started at http://127.10.225.189:35901/ using document root <none> and password file <none>
I20260806 23:54:36.685407 30849 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:36.685468 30849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:36.685647 30849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:54:36.687734 30849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/data/instance:
uuid: "f062abc088fb46a8b697a6e5f0b6496e"
format_stamp: "Formatted at 2026-08-06 23:54:36 on dist-test-slave-nj21"
server_key: "c823c51335df672c3ad197e208b1f978"
server_key_iv: "c3569b7305fb05e3bcb771200f5e6e13"
server_key_version: "encryptionkey@0"
I20260806 23:54:36.688333 30849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/wal/instance:
uuid: "f062abc088fb46a8b697a6e5f0b6496e"
format_stamp: "Formatted at 2026-08-06 23:54:36 on dist-test-slave-nj21"
server_key: "c823c51335df672c3ad197e208b1f978"
server_key_iv: "c3569b7305fb05e3bcb771200f5e6e13"
server_key_version: "encryptionkey@0"
I20260806 23:54:36.692162 30849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 23:54:36.694648 30869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:36.696043 30849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:36.696165 30849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/wal
uuid: "f062abc088fb46a8b697a6e5f0b6496e"
format_stamp: "Formatted at 2026-08-06 23:54:36 on dist-test-slave-nj21"
server_key: "c823c51335df672c3ad197e208b1f978"
server_key_iv: "c3569b7305fb05e3bcb771200f5e6e13"
server_key_version: "encryptionkey@0"
I20260806 23:54:36.696262 30849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:36.724747 30849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:36.728217 30849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:36.728390 30849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:36.736399 30849 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.189:45567
I20260806 23:54:36.736726 30946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.189:45567 every 8 connection(s)
I20260806 23:54:36.737427 30849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
I20260806 23:54:36.739261 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 30849
I20260806 23:54:36.739382 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-1/wal/instance
I20260806 23:54:36.739701 11142 external_mini_cluster.cc:1428] Setting key e209ef391ff54d0610fbbdc8229bd352
I20260806 23:54:36.740523 30947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:36.744386 30947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:54:36.744841 30839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } attempt: 3
WARNING: no policy specified for kudu/127.10.225.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.188@KRBTEST.COM" created.
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:36.758581 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.745406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:45567 (local address 127.0.0.1:37818)
0806 23:54:36.745609 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.745625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:36.745715 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:36.748134 (+ 2419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:36.748138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:36.748322 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:36.748493 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.748501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.750281 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.750285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.750836 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.750845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.751587 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.751591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.751728 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.752418 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:36.752439 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:36.754193 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:36.756594 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.756597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.756600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.756832 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.757164 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.757167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.757169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.757251 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.757919 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:36.757923 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:36.758056 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 23:54:36.758165 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 23:54:36.758338 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"spinlock_wait_cycles":5120}
I20260806 23:54:36.759588 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.747085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.10.225.190:35587)
0806 23:54:36.747198 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.747203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.747593 (+ 390us) server_negotiation.cc:407] Connection header received
0806 23:54:36.748108 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.748112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.748168 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.748265 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.749473 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.750343 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.751723 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.752124 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.752255 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.755119 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:36.755123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:36.755126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:36.755153 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:36.757677 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.758189 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.758193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.758195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.758271 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.758523 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.758527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.758528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.758738 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:36.758829 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:36.759287 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 23:54:36.759426 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":4864}
I20260806 23:54:36.760653 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.747028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35587 (local address 127.0.0.1:48310)
0806 23:54:36.747462 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.747512 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:36.747889 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:36.748340 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:36.748348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:36.748772 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:36.749324 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.749337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.750623 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.750627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.751544 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.751560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.752399 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.752404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.752545 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.753133 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:36.753158 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:36.754933 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:36.757802 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.757811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.757816 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.758090 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.758356 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.758359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.758361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.758441 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.758884 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:36.758891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:36.759053 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 23:54:36.759191 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 23:54:36.759654 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":175,"spinlock_wait_cycles":1280,"thread_start_us":93,"threads_started":1}
I20260806 23:54:36.760684 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.746230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.10.225.189:45567)
0806 23:54:36.746728 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.746742 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.746776 (+ 34us) server_negotiation.cc:407] Connection header received
0806 23:54:36.747649 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.747697 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.747962 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.748657 (+ 695us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.749132 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.750183 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.750953 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.751474 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.751624 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.754347 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:36.754356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:36.754376 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:36.754406 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:36.756479 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.756940 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.756948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.756955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.757071 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.757336 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.757340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.757342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.757806 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:36.757951 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:36.759083 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0806 23:54:36.759580 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":358,"spinlock_wait_cycles":10624,"thread_start_us":153,"threads_started":1}
I20260806 23:54:36.762708 30947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:54:36.763119 30839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:54:36.763876 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.763552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59186)
0806 23:54:36.763733 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.763797 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 23:54:36.764540 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.764307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59190)
0806 23:54:36.764434 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.764483 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 23:54:36.764652 30839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:36065: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
I20260806 23:54:36.771824 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.763355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:45567 (local address 127.0.0.1:37832)
0806 23:54:36.763496 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.763512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:36.763598 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:36.763895 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:36.763898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:36.764180 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:36.764318 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.764326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.765230 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.765234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.765886 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.765896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.766481 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.766484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.766562 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.767120 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:36.767131 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:36.768429 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:36.770207 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.770209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.770211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.770488 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.770799 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.770803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.770805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.770901 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.771285 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:36.771288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:36.771404 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 23:54:36.771506 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 23:54:36.771618 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 23:54:36.771840 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.763563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.10.225.189:45567)
0806 23:54:36.763650 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.763654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.763667 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:54:36.763737 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.763740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.763784 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.763858 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.764455 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.765114 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.766012 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.766384 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.766467 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.768543 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:36.768546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:36.768548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:36.768569 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:36.770095 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.770596 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.770599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.770600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.770678 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.771004 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.771007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.771008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.771194 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:36.771288 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:36.771586 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 23:54:36.771715 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 23:54:36.772606 30947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:54:36.773279 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.773047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59192)
0806 23:54:36.773152 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.773227 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab.
Entry for principal kudu/127.10.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab.
W20260806 23:54:36.773980 30874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
I20260806 23:54:36.774893 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.774680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59208)
0806 23:54:36.774780 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.774833 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 23:54:36.774957 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.188:36065
--webserver_interface=127.10.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab
--principal=kudu/127.10.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35587,127.10.225.189:45567,127.10.225.188:36065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:36.775043 30947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:36065: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
I20260806 23:54:36.795800 30839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } attempt: 1
I20260806 23:54:36.796711 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.796426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59222)
0806 23:54:36.796568 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.796630 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 23:54:36.797407 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.797232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59230)
0806 23:54:36.797315 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.797359 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 23:54:36.797646 30839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:36065: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
I20260806 23:54:36.824332 30947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } attempt: 1
I20260806 23:54:36.825306 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.824964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59238)
0806 23:54:36.825130 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.825202 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 23:54:36.826063 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.825844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59248)
0806 23:54:36.825948 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.825998 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 23:54:36.826225 30947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:36065: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
I20260806 23:54:36.872841 30839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } attempt: 2
I20260806 23:54:36.873927 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.873399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59256)
0806 23:54:36.873756 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.873833 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 23:54:36.874738 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.874502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59266)
0806 23:54:36.874626 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.874680 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 23:54:36.874892 30839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:36065: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
I20260806 23:54:36.894421 30947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } attempt: 2
I20260806 23:54:36.895308 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.895008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59282)
0806 23:54:36.895165 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.895228 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 23:54:36.896015 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.895804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59296)
0806 23:54:36.895905 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.895951 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 23:54:36.896158 30947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:36065: Network error: Client connection negotiation failed: client connection to 127.10.225.188:36065: connect: Connection refused (error 111)
W20260806 23:54:36.896842 30956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:36.897113 30956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:36.897183 30956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:36.901381 30956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:54:36.901445 30956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:36.901474 30956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:36.901506 30956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:54:36.901535 30956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:54:36.907094 30956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35587,127.10.225.189:45567,127.10.225.188:36065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab
--principal=kudu/127.10.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.188:36065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:36.908437 30956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:36.909509 30956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:36.918114 30962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:36.918108 30963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:36.918248 30965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:36.918762 30956 server_base.cc:1034] running on GCE node
I20260806 23:54:36.919361 30956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:36.920549 30956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:36.921732 30956 hybrid_clock.cc:648] HybridClock initialized: now 1786060476921713 us; error 30 us; skew 500 ppm
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:36.924916 30956 init.cc:377] Logged in from keytab as kudu/127.10.225.188@KRBTEST.COM (short username kudu)
I20260806 23:54:36.926369 30956 webserver.cc:460] Webserver started at http://127.10.225.188:38691/ using document root <none> and password file <none>
I20260806 23:54:36.926988 30956 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:36.927059 30956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:36.927259 30956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:54:36.929430 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/data/instance:
uuid: "bb2c30cefb2a45c284955514e3a26372"
format_stamp: "Formatted at 2026-08-06 23:54:36 on dist-test-slave-nj21"
server_key: "23b7a7b839c9585347a2c4cf7ea0d9f7"
server_key_iv: "d40320190a7617537ad70d148ffb6252"
server_key_version: "encryptionkey@0"
I20260806 23:54:36.929996 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/wal/instance:
uuid: "bb2c30cefb2a45c284955514e3a26372"
format_stamp: "Formatted at 2026-08-06 23:54:36 on dist-test-slave-nj21"
server_key: "23b7a7b839c9585347a2c4cf7ea0d9f7"
server_key_iv: "d40320190a7617537ad70d148ffb6252"
server_key_version: "encryptionkey@0"
I20260806 23:54:36.934062 30956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 23:54:36.936684 30974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:36.938342 30956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:36.938609 30956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/wal
uuid: "bb2c30cefb2a45c284955514e3a26372"
format_stamp: "Formatted at 2026-08-06 23:54:36 on dist-test-slave-nj21"
server_key: "23b7a7b839c9585347a2c4cf7ea0d9f7"
server_key_iv: "d40320190a7617537ad70d148ffb6252"
server_key_version: "encryptionkey@0"
I20260806 23:54:36.938740 30956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:36.954774 30956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:36.958503 30956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:36.958709 30956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:36.965976 30956 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.188:36065
I20260806 23:54:36.965996 31056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.188:36065 every 8 connection(s)
I20260806 23:54:36.967015 30956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
I20260806 23:54:36.970847 31058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:36.973124 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 30956
I20260806 23:54:36.973273 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786060299689793-11142-0/minicluster-data/master-2/wal/instance
I20260806 23:54:36.973541 11142 external_mini_cluster.cc:1428] Setting key 099d8d9213e372796d88eee5548af3dd
I20260806 23:54:36.974080 30839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } attempt: 3
I20260806 23:54:36.975174 31058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
I20260806 23:54:36.987317 30947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } attempt: 3
I20260806 23:54:36.988298 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.977517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.10.225.190:35587)
0806 23:54:36.977628 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.977632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.978010 (+ 378us) server_negotiation.cc:407] Connection header received
0806 23:54:36.978118 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.978120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.978163 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.978247 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.979003 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.979498 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.980448 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.980768 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.980862 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.984011 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:36.984016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:36.984047 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:36.984078 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:36.985624 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.986108 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.986110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.986112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.986182 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.986567 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.986571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.986573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.986911 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:36.987040 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:36.987953 (+ 913us) server_negotiation.cc:299] Negotiation successful
0806 23:54:36.988082 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260806 23:54:36.989748 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.974529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59312)
0806 23:54:36.974716 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.974732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:36.974840 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:36.976887 (+ 2047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:36.976891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:36.977113 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:36.977246 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.977254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.979061 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.979064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.979589 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.979598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.980565 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.980569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.980978 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.982073 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:36.982093 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:36.985153 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:36.987714 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.987718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.987720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.987981 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.988342 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.988345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.988347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.988463 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.989247 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:36.989251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:36.989344 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 23:54:36.989426 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 23:54:36.989591 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":325}
I20260806 23:54:36.990015 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.977116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35587 (local address 127.0.0.1:48316)
0806 23:54:36.977870 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.977916 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:36.978038 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:36.978301 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:36.978306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:36.978724 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:36.978855 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.978869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.979608 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.979611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.980335 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.980345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.980944 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.980947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.981051 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.981762 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:36.981784 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:36.983846 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:36.985726 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.985736 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.985748 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.986013 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.986264 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.986267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.986269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.986441 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.987218 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:36.987226 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:36.987592 (+ 366us) client_negotiation.cc:769] Sending connection context
0806 23:54:36.988519 (+ 927us) client_negotiation.cc:240] Negotiation successful
0806 23:54:36.988871 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260806 23:54:36.990935 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.974953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.10.225.188:36065)
0806 23:54:36.975454 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.975470 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.975506 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:54:36.976230 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.976250 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.976606 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.977358 (+ 752us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.977465 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.978898 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.979723 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.980445 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.980604 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.985332 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:36.985341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:36.985357 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:36.985386 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:36.987558 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.988121 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.988127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.988129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.988216 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.988824 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.988827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.988829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.989132 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:36.990412 (+ 1280us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:36.990532 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 23:54:36.990679 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":327,"server-negotiator.queue_time_us":365,"spinlock_wait_cycles":4864,"thread_start_us":124,"threads_started":1}
I20260806 23:54:36.990967 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.974953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.10.225.188:36065)
0806 23:54:36.975475 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.975483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.975507 (+ 24us) server_negotiation.cc:407] Connection header received
0806 23:54:36.976258 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.976263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.976609 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.976909 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.977669 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.978875 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.980660 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.980987 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.981124 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.986183 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:36.986187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:36.986190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:36.986216 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:36.988464 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.988976 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.988979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.988981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.989059 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.989362 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.989364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.989366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.989577 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:36.990542 (+ 965us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:36.990622 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 23:54:36.990758 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":432,"spinlock_wait_cycles":11008,"thread_start_us":152,"threads_started":1}
I20260806 23:54:36.991513 31058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
I20260806 23:54:36.996034 30839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8: Bootstrap starting.
I20260806 23:54:36.997866 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.988240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59314)
0806 23:54:36.988460 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.988475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:36.988566 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:36.989080 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:36.989084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:36.989268 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:36.989384 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.989392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.990473 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.990477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.991293 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.991305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.991840 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.991844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.991926 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.992528 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:36.992542 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:36.993990 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:36.996266 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.996269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.996271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.996516 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.996810 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:36.996813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:36.996815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:36.996912 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:36.997383 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:36.997386 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:36.997485 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 23:54:36.997566 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 23:54:36.997699 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
I20260806 23:54:36.998538 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.987917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.10.225.188:36065)
0806 23:54:36.988832 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.988843 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.988854 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:54:36.988884 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.988886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.988929 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.989503 (+ 574us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.989597 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.990350 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.991434 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.991711 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.992154 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.994542 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:36.994546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:36.994549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:36.994571 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:36.996173 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.996625 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.996628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.996630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.996710 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:36.997029 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:36.997031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:36.997033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:36.997251 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:36.998165 (+ 914us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:36.998237 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 23:54:36.998362 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":140,"threads_started":1}
I20260806 23:54:36.999086 30839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8: Neither blocks nor log segments found. Creating new log.
Aug 06 23:54:36 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:37.000754 30839 log.cc:824] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:37.001816 30947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e: Bootstrap starting.
I20260806 23:54:37.002914 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.992432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.10.225.189:45567)
0806 23:54:36.993223 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.993227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.993242 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:54:36.993275 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.993278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.993319 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.993397 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.993954 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.994597 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.996174 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.996493 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.996568 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.999013 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:36.999017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:36.999019 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:36.999043 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:37.001126 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.001601 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.001605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.001607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.001688 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.002011 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.002014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.002016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.002244 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:37.002314 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:37.002655 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 23:54:37.002763 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708}
I20260806 23:54:37.003146 30839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8: No bootstrap required, opened a new log
I20260806 23:54:37.003362 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.992318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:45567 (local address 127.0.0.1:37844)
0806 23:54:36.992573 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:36.992587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:36.992670 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:36.993516 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:36.993520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:36.993696 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:36.993809 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.993829 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.994723 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.994726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.996036 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:36.996049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.996701 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.996704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:36.996798 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:36.997239 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:36.997251 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:36.998867 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:37.001252 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:37.001255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:37.001258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:37.001493 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:37.001798 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:37.001801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:37.001803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:37.001906 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:37.002389 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:37.002391 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:37.002484 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 23:54:37.003036 (+ 552us) client_negotiation.cc:240] Negotiation successful
0806 23:54:37.003187 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"spinlock_wait_cycles":7040,"thread_start_us":104,"threads_started":1}
I20260806 23:54:37.003887 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:36.994144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.10.225.190:35587)
0806 23:54:36.994248 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:36.994251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:36.994263 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:54:36.994295 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:36.994298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:36.994347 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:36.994418 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:36.995153 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.995853 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.996883 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:36.997184 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:36.997284 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:37.000062 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:37.000067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:37.000069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:37.000094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:37.001667 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.002159 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.002163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.002164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.002251 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.002542 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.002546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.002547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.002751 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:37.002910 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:37.003588 (+ 678us) server_negotiation.cc:299] Negotiation successful
0806 23:54:37.003738 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:54:37.004460 31058 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:54:37.005087 30947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:37.006183 30947 log.cc:824] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:37.008526 30839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } }
I20260806 23:54:37.008881 30839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:37.009006 30839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cee1ac085fd4934ace8328ef74247a8, State: Initialized, Role: FOLLOWER
Aug 06 23:54:37 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
I20260806 23:54:37.009657 30839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } }
I20260806 23:54:37.009912 30947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e: No bootstrap required, opened a new log
Aug 06 23:54:37 dist-test-slave-nj21 krb5kdc[30723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
I20260806 23:54:37.011336 30839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:54:37.013499 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:37.004919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:36065 (local address 127.0.0.1:59316)
0806 23:54:37.005133 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:37.005146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:37.005217 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:37.005683 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:37.005686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:37.005858 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:37.005965 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:37.005973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.006872 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.006874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:37.007371 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:37.007377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.008013 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.008016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:37.008107 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:37.008519 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:37.008533 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:37.010055 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:37.011890 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:37.011893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:37.011895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:37.012125 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:37.012398 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:37.012400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:37.012402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:37.012488 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:37.012890 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:37.012893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:37.013001 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 23:54:37.013121 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 23:54:37.013275 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260806 23:54:37.013533 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:37.005135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.10.225.188:36065)
0806 23:54:37.005496 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:37.005500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:37.005512 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:54:37.005544 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:37.005547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:37.005585 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:37.005663 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:37.006082 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.006774 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.007473 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.007810 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.007910 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:37.010180 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:37.010183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:37.010185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:37.010207 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:37.011780 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.012217 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.012220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.012222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.012292 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.012576 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.012579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.012580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.012779 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:37.012874 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:37.013191 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 23:54:37.013362 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":290}
I20260806 23:54:37.014384 31072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.014667 31072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:54:37.015146 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:37.005174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.10.225.189:45567)
0806 23:54:37.005277 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:37.005281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:37.005294 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:54:37.005406 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:37.005408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:37.005453 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:37.005534 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:37.006352 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.007066 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.008198 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.008494 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.008567 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:37.011485 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:37.011488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:37.011490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:37.011515 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:37.013112 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.013554 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.013557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.013559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.013640 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.013918 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.013921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.013923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.014121 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:37.014306 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:37.014934 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 23:54:37.015041 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:54:37.015146 30947 raft_consensus.cc:348] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } }
I20260806 23:54:37.015404 30947 raft_consensus.cc:374] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:37.015478 30947 raft_consensus.cc:729] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f062abc088fb46a8b697a6e5f0b6496e, State: Initialized, Role: FOLLOWER
I20260806 23:54:37.015973 30947 consensus_queue.cc:260] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } }
I20260806 23:54:37.016588 31058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372: Bootstrap starting.
I20260806 23:54:37.017326 30947 sys_catalog.cc:564] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:54:37.019147 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:54:37.019430 31058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:37.020363 31058 log.cc:824] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372: Log is configured to *not* fsync() on all Append() calls
W20260806 23:54:37.021625 31094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:54:37.021696 31094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:54:37.022531 31058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372: No bootstrap required, opened a new log
I20260806 23:54:37.020025 30739 master_runner.cc:186] Error getting master registration for 127.10.225.189:45567: OK, instance_id { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" instance_seqno: 1786060476734272 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 23:54:37.024561 30739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:54:37.026743 31086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.026892 31086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:54:37.027372 31058 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } }
I20260806 23:54:37.027628 31058 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:37.027704 31058 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2c30cefb2a45c284955514e3a26372, State: Initialized, Role: FOLLOWER
I20260806 23:54:37.028224 31058 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } }
W20260806 23:54:37.028844 31121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:54:37.028926 31121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:54:37.029631 31058 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 23:54:37.036193 31145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:54:37.036295 31145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:54:37.041606 30956 master_runner.cc:263] No leader master found from master bb2c30cefb2a45c284955514e3a26372
I20260806 23:54:37.041683 30956 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:54:37.031737 30849 master_runner.cc:186] Error getting master registration for 127.10.225.188:36065: OK, instance_id { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" instance_seqno: 1786060476964006 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 23:54:37.043267 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:37.025584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.10.225.188:36065)
0806 23:54:37.025697 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:37.025702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:37.033389 (+ 7687us) server_negotiation.cc:407] Connection header received
0806 23:54:37.033625 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:37.033629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:37.033682 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:37.033732 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:37.035131 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.036027 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.036974 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.037294 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.037391 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:37.039239 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:37.039244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:37.039247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:37.039273 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:37.040976 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.041770 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.041774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.041776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.041896 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.042234 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.042238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.042240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.042475 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:37.042581 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:37.042973 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 23:54:37.043106 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:54:37.044188 30849 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:54:37.032536 31126 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.047356 31126 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:54:37.052027 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:37.026679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.10.225.190:35587)
0806 23:54:37.026787 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:37.026791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:37.037168 (+ 10377us) server_negotiation.cc:407] Connection header received
0806 23:54:37.037340 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:37.037343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:37.037388 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:37.037460 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:37.044344 (+ 6884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.045143 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.046309 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.046668 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.046740 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:37.048558 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:37.048563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:37.048566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:37.048593 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:37.050201 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.050654 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.050657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.050659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.050747 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.051040 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.051043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.051045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.051259 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:37.051365 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:37.051712 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 23:54:37.051851 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 23:54:37.059517 31126 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:54:37.060672 31126 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } }
I20260806 23:54:37.062533 30800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb2c30cefb2a45c284955514e3a26372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cee1ac085fd4934ace8328ef74247a8" is_pre_election: true
I20260806 23:54:37.062891 30800 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb2c30cefb2a45c284955514e3a26372 in term 0.
I20260806 23:54:37.063477 30975 leader_election.cc:302] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cee1ac085fd4934ace8328ef74247a8, bb2c30cefb2a45c284955514e3a26372; no voters:
I20260806 23:54:37.064664 30911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb2c30cefb2a45c284955514e3a26372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f062abc088fb46a8b697a6e5f0b6496e" is_pre_election: true
I20260806 23:54:37.064973 30911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb2c30cefb2a45c284955514e3a26372 in term 0.
I20260806 23:54:37.065448 31126 leader_election.cc:288] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cee1ac085fd4934ace8328ef74247a8 (127.10.225.190:35587), f062abc088fb46a8b697a6e5f0b6496e (127.10.225.189:45567)
I20260806 23:54:37.065761 31126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:54:37.065900 31126 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:54:37.065999 31126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:37.067519 31126 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } }
I20260806 23:54:37.068310 30800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb2c30cefb2a45c284955514e3a26372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cee1ac085fd4934ace8328ef74247a8"
I20260806 23:54:37.068461 30800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:37.069818 30800 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb2c30cefb2a45c284955514e3a26372 in term 1.
I20260806 23:54:37.070220 30975 leader_election.cc:302] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cee1ac085fd4934ace8328ef74247a8, bb2c30cefb2a45c284955514e3a26372; no voters:
I20260806 23:54:37.070895 30911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb2c30cefb2a45c284955514e3a26372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f062abc088fb46a8b697a6e5f0b6496e"
I20260806 23:54:37.071038 30911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:37.071777 31147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:54:37.072046 31147 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [term 1 LEADER]: Becoming Leader. State: Replica: bb2c30cefb2a45c284955514e3a26372, State: Running, Role: LEADER
I20260806 23:54:37.072456 30911 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb2c30cefb2a45c284955514e3a26372 in term 1.
I20260806 23:54:37.072458 31147 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } }
I20260806 23:54:37.072937 31126 leader_election.cc:288] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [CANDIDATE]: Term 1 election: Requested vote from peers 3cee1ac085fd4934ace8328ef74247a8 (127.10.225.190:35587), f062abc088fb46a8b697a6e5f0b6496e (127.10.225.189:45567)
I20260806 23:54:37.075197 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:37.026420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.10.225.189:45567)
0806 23:54:37.026596 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:37.026600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:37.034995 (+ 8395us) server_negotiation.cc:407] Connection header received
0806 23:54:37.035599 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:37.035602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:37.035653 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:37.035791 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:37.054383 (+ 18592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.055217 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.056446 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:37.056786 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:37.056872 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:37.058622 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:37.058627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:37.058630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:37.058659 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:37.060314 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.073731 (+ 13417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.073736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.073738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.073856 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:37.074152 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:37.074155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:37.074157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:37.074397 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:37.074514 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:37.074897 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 23:54:37.075035 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 23:54:37.075107 31126 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb2c30cefb2a45c284955514e3a26372. Latest consensus state: current_term: 1 leader_uuid: "bb2c30cefb2a45c284955514e3a26372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.075242 31126 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:37.079977 31151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:54:37.081038 31151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:54:37.092034 30911 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [term 1 FOLLOWER]: Refusing update from remote peer bb2c30cefb2a45c284955514e3a26372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:54:37.094990 31126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:54:37.100646 31086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb2c30cefb2a45c284955514e3a26372. Latest consensus state: current_term: 1 leader_uuid: "bb2c30cefb2a45c284955514e3a26372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.100814 31086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:54:37.101253 30800 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [term 1 FOLLOWER]: Refusing update from remote peer bb2c30cefb2a45c284955514e3a26372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:54:37.102095 31126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:54:37.106252 31072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb2c30cefb2a45c284955514e3a26372. Latest consensus state: current_term: 1 leader_uuid: "bb2c30cefb2a45c284955514e3a26372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.106400 31072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:54:37.118372 31147 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb2c30cefb2a45c284955514e3a26372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.118619 31147 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:37.119056 31152 mvcc.cc:205] Tried to move back new op lower bound from 7315703714160033792 to 7315703714092568576. Current Snapshot: MvccSnapshot[applied={T|T < 7315703714160033792}]
I20260806 23:54:37.120467 31072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb2c30cefb2a45c284955514e3a26372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.120620 31072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:54:37.123879 31126 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb2c30cefb2a45c284955514e3a26372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.124035 31126 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372 [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:37.125331 31086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb2c30cefb2a45c284955514e3a26372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cee1ac085fd4934ace8328ef74247a8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35587 } } peers { permanent_uuid: "f062abc088fb46a8b697a6e5f0b6496e" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 45567 } } peers { permanent_uuid: "bb2c30cefb2a45c284955514e3a26372" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 36065 } } }
I20260806 23:54:37.125478 31086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:54:37.125922 31151 catalog_manager.cc:1347] Generated new cluster ID: d03a273604714d509fe001ce04d49055
I20260806 23:54:37.125990 31151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:54:37.126781 31156 mvcc.cc:205] Tried to move back new op lower bound from 7315703714160033792 to 7315703714092568576. Current Snapshot: MvccSnapshot[applied={T|T < 7315703714160033792}]
I20260806 23:54:37.141738 31151 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:54:37.144640 31151 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:54:37.157261 31151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb2c30cefb2a45c284955514e3a26372: Generated new TSK 0
I20260806 23:54:37.158152 31151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:54:38.023134 31094 catalog_manager.cc:1259] Loaded cluster ID: d03a273604714d509fe001ce04d49055
I20260806 23:54:38.023231 31094 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8: loading cluster ID for follower catalog manager: success
I20260806 23:54:38.025161 31094 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8: acquiring CA information for follower catalog manager: success
I20260806 23:54:38.026394 31094 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3cee1ac085fd4934ace8328ef74247a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:54:38.030522 31121 catalog_manager.cc:1259] Loaded cluster ID: d03a273604714d509fe001ce04d49055
I20260806 23:54:38.030582 31121 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e: loading cluster ID for follower catalog manager: success
I20260806 23:54:38.032351 31121 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e: acquiring CA information for follower catalog manager: success
I20260806 23:54:38.033497 31121 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f062abc088fb46a8b697a6e5f0b6496e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:54:38.236224 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 30739
I20260806 23:54:38.251348 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 30849
W20260806 23:54:38.262224 30977 connection.cc:535] client connection to 127.10.225.189:45567 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 23:54:38.262485 30977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 23:54:38.262522 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 30956
2026-08-06T23:54:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3277 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:54:38 dist-test-slave-nj21 krb5kdc[31169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:54:38 dist-test-slave-nj21 krb5kdc[31169](info): set up 2 sockets
Aug 06 23:54:38 dist-test-slave-nj21 krb5kdc[31169](info): commencing operation
krb5kdc: starting...
W20260806 23:54:39.421013 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:54:39 dist-test-slave-nj21 krb5kdc[31169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:54:39.555135 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:44945
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:39.702304 31192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:39.702870 31192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:39.703006 31192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:39.707986 31192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:54:39.708062 31192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:39.708081 31192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:39.708098 31192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:54:39.708113 31192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:54:39.714234 31192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:44945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:39.715430 31192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:39.716544 31192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:39.725368 31197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:39.725466 31198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:39.725502 31205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:39.725685 31192 server_base.cc:1034] running on GCE node
I20260806 23:54:39.726331 31192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:39.727478 31192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:39.728654 31192 hybrid_clock.cc:648] HybridClock initialized: now 1786060479728644 us; error 58 us; skew 500 ppm
Aug 06 23:54:39 dist-test-slave-nj21 krb5kdc[31169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060479, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:39.731493 31192 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:54:39.732452 31192 webserver.cc:460] Webserver started at http://127.10.225.190:38287/ using document root <none> and password file <none>
I20260806 23:54:39.733012 31192 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:39.733071 31192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:39.733294 31192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:54:39.735373 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "5486b0e85a9247eeb8ce437944a01ff8"
format_stamp: "Formatted at 2026-08-06 23:54:39 on dist-test-slave-nj21"
server_key: "40bdface9d03f03f61dc614d54aa123a"
server_key_iv: "263c9692741fd34a8e134d565967a935"
server_key_version: "encryptionkey@0"
I20260806 23:54:39.735973 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "5486b0e85a9247eeb8ce437944a01ff8"
format_stamp: "Formatted at 2026-08-06 23:54:39 on dist-test-slave-nj21"
server_key: "40bdface9d03f03f61dc614d54aa123a"
server_key_iv: "263c9692741fd34a8e134d565967a935"
server_key_version: "encryptionkey@0"
I20260806 23:54:39.740134 31192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:54:39.742749 31215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:39.744166 31192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:39.744298 31192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "5486b0e85a9247eeb8ce437944a01ff8"
format_stamp: "Formatted at 2026-08-06 23:54:39 on dist-test-slave-nj21"
server_key: "40bdface9d03f03f61dc614d54aa123a"
server_key_iv: "263c9692741fd34a8e134d565967a935"
server_key_version: "encryptionkey@0"
I20260806 23:54:39.744407 31192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:39.761931 31192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:39.765446 31192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:39.765627 31192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:39.772956 31192 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:44945
I20260806 23:54:39.772995 31292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:44945 every 8 connection(s)
I20260806 23:54:39.773962 31192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:54:39.774130 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 31192
I20260806 23:54:39.774272 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:54:39.774622 11142 external_mini_cluster.cc:1428] Setting key 6a97d0e4b729da154bf64b677e803810
I20260806 23:54:39.777248 31294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:39.783381 31294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8: Bootstrap starting.
Aug 06 23:54:39 dist-test-slave-nj21 krb5kdc[31169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:39.785756 31294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:39.786504 31294 log.cc:824] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:39.788502 31294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8: No bootstrap required, opened a new log
I20260806 23:54:39.789639 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:39.776035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.10.225.190:44945)
0806 23:54:39.776707 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:39.776731 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:39.776795 (+ 64us) server_negotiation.cc:407] Connection header received
0806 23:54:39.777724 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:39.777748 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:39.778214 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:39.778644 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:39.779625 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:39.780847 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:39.781974 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:39.782505 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:39.782670 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:39.785229 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:39.785235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:39.785249 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:39.785290 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:39.786920 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:39.787327 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:39.787333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:39.787338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:39.787447 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:39.787721 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:39.787724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:39.787725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:39.788131 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:39.788317 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:39.788593 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 23:54:39.788902 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":135,"threads_started":1}
I20260806 23:54:39.791666 31294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5486b0e85a9247eeb8ce437944a01ff8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44945 } }
I20260806 23:54:39.791970 31294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:39.792033 31294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5486b0e85a9247eeb8ce437944a01ff8, State: Initialized, Role: FOLLOWER
I20260806 23:54:39.792466 31294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5486b0e85a9247eeb8ce437944a01ff8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44945 } }
I20260806 23:54:39.792604 31294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:54:39.792672 31294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:54:39.792786 31294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:39.792811 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:54:39.793794 31294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5486b0e85a9247eeb8ce437944a01ff8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44945 } }
I20260806 23:54:39.794121 31294 leader_election.cc:302] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5486b0e85a9247eeb8ce437944a01ff8; no voters:
I20260806 23:54:39.794464 31294 leader_election.cc:288] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:54:39.794543 31302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:54:39.794750 31302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [term 1 LEADER]: Becoming Leader. State: Replica: 5486b0e85a9247eeb8ce437944a01ff8, State: Running, Role: LEADER
I20260806 23:54:39.795083 31302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5486b0e85a9247eeb8ce437944a01ff8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44945 } }
I20260806 23:54:39.795645 31294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:54:39.796845 31305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5486b0e85a9247eeb8ce437944a01ff8. Latest consensus state: current_term: 1 leader_uuid: "5486b0e85a9247eeb8ce437944a01ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5486b0e85a9247eeb8ce437944a01ff8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44945 } } }
I20260806 23:54:39.796841 31304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5486b0e85a9247eeb8ce437944a01ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5486b0e85a9247eeb8ce437944a01ff8" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44945 } } }
I20260806 23:54:39.797000 31305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:39.797000 31304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8 [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:39.797448 31314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:54:39.800678 31314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:54:39.807511 31314 catalog_manager.cc:1347] Generated new cluster ID: 7fe00354169b42a784e68cb1d85d406a
I20260806 23:54:39.807593 31314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:54:39.823637 31314 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:54:39.825071 31314 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:54:39.836274 31314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5486b0e85a9247eeb8ce437944a01ff8: Generated new TSK 0
I20260806 23:54:39.837138 31314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:54:39.849373 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 31192
2026-08-06T23:54:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1587 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:54:39 dist-test-slave-nj21 krb5kdc[31331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:54:39 dist-test-slave-nj21 krb5kdc[31331](info): set up 2 sockets
Aug 06 23:54:39 dist-test-slave-nj21 krb5kdc[31331](info): commencing operation
krb5kdc: starting...
W20260806 23:54:40.999034 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:54:41 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:54:41.139731 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34845
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:41.261319 31349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:41.261618 31349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:41.261662 31349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:41.265843 31349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:54:41.265929 31349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:41.265949 31349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:41.265964 31349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:54:41.265978 31349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:54:41.271533 31349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:41.272743 31349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:41.273787 31349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:41.282377 31361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:41.282438 31349 server_base.cc:1034] running on GCE node
W20260806 23:54:41.282367 31355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:41.282371 31357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:41.283205 31349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:41.284401 31349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:41.285598 31349 hybrid_clock.cc:648] HybridClock initialized: now 1786060481285569 us; error 42 us; skew 500 ppm
Aug 06 23:54:41 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:41.291505 31349 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:54:41.292564 31349 webserver.cc:460] Webserver started at http://127.10.225.190:34315/ using document root <none> and password file <none>
I20260806 23:54:41.293164 31349 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:41.293228 31349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:41.293429 31349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:54:41.295567 31349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "1c0e22cc086c45d7a023d019fb695d8e"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "759821544b1c1e530156853b737236c5"
server_key_iv: "332c0a714b645d938c0338e8d617b426"
server_key_version: "encryptionkey@0"
I20260806 23:54:41.296180 31349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "1c0e22cc086c45d7a023d019fb695d8e"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "759821544b1c1e530156853b737236c5"
server_key_iv: "332c0a714b645d938c0338e8d617b426"
server_key_version: "encryptionkey@0"
I20260806 23:54:41.300063 31349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:54:41.302788 31368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:41.304332 31349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:41.304464 31349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "1c0e22cc086c45d7a023d019fb695d8e"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "759821544b1c1e530156853b737236c5"
server_key_iv: "332c0a714b645d938c0338e8d617b426"
server_key_version: "encryptionkey@0"
I20260806 23:54:41.304580 31349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:41.329794 31349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:41.334079 31349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:41.334345 31349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:41.342681 31349 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:34845
I20260806 23:54:41.342949 31439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:34845 every 8 connection(s)
I20260806 23:54:41.344040 31349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:54:41.347353 31440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:41.349002 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 31349
I20260806 23:54:41.349143 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:54:41.349422 11142 external_mini_cluster.cc:1428] Setting key 5fb20b7e613634792b7caf1159581cef
I20260806 23:54:41.353907 31440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Bootstrap starting.
I20260806 23:54:41.356555 31440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:41.357841 31440 log.cc:824] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:54:41 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:41.360106 31440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: No bootstrap required, opened a new log
I20260806 23:54:41.363734 31440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:41.363962 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:41.350817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.10.225.190:34845)
0806 23:54:41.351251 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:41.351267 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:41.351304 (+ 37us) server_negotiation.cc:407] Connection header received
0806 23:54:41.352008 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:41.352027 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:41.352357 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:41.352625 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:41.353560 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:41.354609 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:41.355782 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:41.356279 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:41.356456 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:41.359271 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:41.359277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:41.359292 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:41.359322 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:41.361017 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:41.361473 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:41.361478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:41.361483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:41.361605 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:41.361886 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:41.361890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:41.361892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:41.362312 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:41.362503 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:41.362827 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 23:54:41.363123 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":125,"threads_started":1}
I20260806 23:54:41.363967 31440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:41.364070 31440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e22cc086c45d7a023d019fb695d8e, State: Initialized, Role: FOLLOWER
I20260806 23:54:41.364570 31440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:41.364738 31440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:54:41.364802 31440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:54:41.364893 31440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:41.365839 31440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:41.366207 31440 leader_election.cc:302] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c0e22cc086c45d7a023d019fb695d8e; no voters:
I20260806 23:54:41.366534 31440 leader_election.cc:288] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:54:41.366659 31448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:54:41.366868 31448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 1 LEADER]: Becoming Leader. State: Replica: 1c0e22cc086c45d7a023d019fb695d8e, State: Running, Role: LEADER
I20260806 23:54:41.367218 31448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:41.367787 31440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:54:41.368840 31449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c0e22cc086c45d7a023d019fb695d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } } }
I20260806 23:54:41.368975 31449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:41.369282 31450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0e22cc086c45d7a023d019fb695d8e. Latest consensus state: current_term: 1 leader_uuid: "1c0e22cc086c45d7a023d019fb695d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } } }
I20260806 23:54:41.369421 31450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:41.369347 31461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:54:41.372865 31461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:54:41.378610 31461 catalog_manager.cc:1347] Generated new cluster ID: 8874634a975d4d95bafe3b1aa40827fb
I20260806 23:54:41.378697 31461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:54:41.392650 31461 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:54:41.393643 31461 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:54:41.404894 31461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Generated new TSK 0
I20260806 23:54:41.405929 31461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:54:41.438576 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34845
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:41.688709 31479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:41.689410 31479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:41.689549 31479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:41.697883 31479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:41.698019 31479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:41.698119 31479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:54:41.708814 31479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:41.711028 31479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:41.713213 31479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:41.731835 31489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:41.731869 31491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:41.731891 31487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:41.732973 31479 server_base.cc:1034] running on GCE node
I20260806 23:54:41.733851 31479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:41.735253 31479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:41.738631 31479 hybrid_clock.cc:648] HybridClock initialized: now 1786060481738602 us; error 90 us; skew 500 ppm
Aug 06 23:54:41 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:41.742233 31479 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:54:41.743280 31479 webserver.cc:460] Webserver started at http://127.10.225.129:39727/ using document root <none> and password file <none>
I20260806 23:54:41.743891 31479 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:41.743955 31479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:41.744186 31479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:54:41.746531 31479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "99315324fc26442398b7cc25f29dafdc"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "47075a54a51e19ee713c4b3f2b093366"
server_key_iv: "6ef231a747360e4906a6b2bbbddf5d70"
server_key_version: "encryptionkey@0"
I20260806 23:54:41.747088 31479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "99315324fc26442398b7cc25f29dafdc"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "47075a54a51e19ee713c4b3f2b093366"
server_key_iv: "6ef231a747360e4906a6b2bbbddf5d70"
server_key_version: "encryptionkey@0"
I20260806 23:54:41.751130 31479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 23:54:41.753645 31500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:41.754925 31479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:54:41.755049 31479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "99315324fc26442398b7cc25f29dafdc"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "47075a54a51e19ee713c4b3f2b093366"
server_key_iv: "6ef231a747360e4906a6b2bbbddf5d70"
server_key_version: "encryptionkey@0"
I20260806 23:54:41.755156 31479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:41.776289 31479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:41.779874 31479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:41.780069 31479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:41.780726 31479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:41.781646 31479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:41.781695 31479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:41.781738 31479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:41.781769 31479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:41.791491 31479 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:43375
I20260806 23:54:41.791517 31659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:43375 every 8 connection(s)
I20260806 23:54:41.792454 31479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:54:41.794155 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 31479
I20260806 23:54:41.794283 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:54:41.794569 11142 external_mini_cluster.cc:1428] Setting key 6d2d707e8f3433c45b1661150123194c
Aug 06 23:54:41 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:54:41.807798 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:41.794478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:55367 (local address 127.10.225.190:34845)
0806 23:54:41.794649 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:41.794653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:41.795579 (+ 926us) server_negotiation.cc:407] Connection header received
0806 23:54:41.796984 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:41.796988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:41.797047 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:41.797127 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:41.798290 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:41.798818 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:41.800015 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:41.800249 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:41.800337 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:41.802995 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:41.803000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:41.803002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:41.803027 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:41.804883 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:41.805418 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:41.805422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:41.805424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:41.805528 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:41.805875 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:41.805878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:41.805880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:41.806075 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:41.806162 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:41.807275 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0806 23:54:41.807456 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 23:54:41.808758 31662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:41.794907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:55367)
0806 23:54:41.795424 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:41.795476 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:41.796767 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:41.797275 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:41.797281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:41.797645 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:41.798137 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:41.798151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:41.798952 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:41.798955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:41.799896 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:41.799907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:41.800402 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:41.800407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:41.800569 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:41.801128 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:41.801145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:41.802786 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:41.805038 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:41.805044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:41.805058 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:41.805320 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:41.805628 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:41.805631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:41.805633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:41.805781 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:41.806182 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:41.806189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:41.806830 (+ 641us) client_negotiation.cc:769] Sending connection context
0806 23:54:41.807173 (+ 343us) client_negotiation.cc:240] Negotiation successful
0806 23:54:41.807841 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":126,"threads_started":1}
I20260806 23:54:41.810431 31660 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34845
I20260806 23:54:41.810673 31660 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:41.811216 31660 heartbeater.cc:511] Master 127.10.225.190:34845 requested a full tablet report, sending...
I20260806 23:54:41.812973 31390 ts_manager.cc:194] Registered new tserver with Master: 99315324fc26442398b7cc25f29dafdc (127.10.225.129:43375)
I20260806 23:54:41.814640 31390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:55367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:54:41.824908 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34845
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:41.947300 31667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:41.947609 31667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:41.947718 31667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:41.953287 31667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:41.953369 31667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:41.953459 31667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:54:41.959367 31667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:41.960904 31667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:41.962456 31667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:41.973672 31679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:41.973702 31676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:41.973683 31675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:41.973896 31667 server_base.cc:1034] running on GCE node
I20260806 23:54:41.974478 31667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:41.975173 31667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:41.976406 31667 hybrid_clock.cc:648] HybridClock initialized: now 1786060481976389 us; error 28 us; skew 500 ppm
Aug 06 23:54:41 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:41.979275 31667 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:54:41.980264 31667 webserver.cc:460] Webserver started at http://127.10.225.130:39225/ using document root <none> and password file <none>
I20260806 23:54:41.980850 31667 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:41.980917 31667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:41.981124 31667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:54:41.983227 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "83052a98317a496ab42ba3b97f224ca9"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "a932a9ce2f9b850e0541519e3c0053f9"
server_key_iv: "2526c2894d066463fc56f01f25013256"
server_key_version: "encryptionkey@0"
I20260806 23:54:41.983853 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "83052a98317a496ab42ba3b97f224ca9"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "a932a9ce2f9b850e0541519e3c0053f9"
server_key_iv: "2526c2894d066463fc56f01f25013256"
server_key_version: "encryptionkey@0"
I20260806 23:54:41.988221 31667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:54:41.990794 31688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:41.992200 31667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:41.992326 31667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "83052a98317a496ab42ba3b97f224ca9"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "a932a9ce2f9b850e0541519e3c0053f9"
server_key_iv: "2526c2894d066463fc56f01f25013256"
server_key_version: "encryptionkey@0"
I20260806 23:54:41.992441 31667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:42.025746 31667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:42.029476 31667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:42.029700 31667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:42.030361 31667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:42.031258 31667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:42.031309 31667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:42.031366 31667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:42.031394 31667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:42.040512 31667 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:45691
I20260806 23:54:42.040539 31863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:45691 every 8 connection(s)
I20260806 23:54:42.041448 31667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:54:42.044152 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 31667
I20260806 23:54:42.044355 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:54:42.044667 11142 external_mini_cluster.cc:1428] Setting key 831883e405b1af242f6b7bb4162a79d3
Aug 06 23:54:42 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060481, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:54:42.059753 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.043924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:33877 (local address 127.10.225.190:34845)
0806 23:54:42.044159 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:42.044164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:42.045551 (+ 1387us) server_negotiation.cc:407] Connection header received
0806 23:54:42.047568 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:42.047572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:42.047634 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:42.047777 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:42.049399 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.050151 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.052075 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.052315 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.052404 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.055867 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:42.055871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:42.055873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:42.055902 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:42.057340 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:42.057868 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:42.057871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:42.057872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:42.057951 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:42.058445 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:42.058448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:42.058449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:42.058659 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:42.058740 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:42.059392 (+ 652us) server_negotiation.cc:299] Negotiation successful
0806 23:54:42.059538 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
I20260806 23:54:42.060696 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.044563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:33877)
0806 23:54:42.045295 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:42.045362 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:42.047161 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:42.047991 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:42.047999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:42.048461 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:42.049171 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.049190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.050362 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.050368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.051929 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.051947 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.052489 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.052492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.052706 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.053502 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:42.053525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:42.055593 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:42.057456 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:42.057464 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:42.057479 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:42.057746 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:42.058119 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:42.058123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:42.058125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:42.058334 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:42.058784 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:42.058790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:42.059079 (+ 289us) client_negotiation.cc:769] Sending connection context
0806 23:54:42.059261 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 23:54:42.059650 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":197,"threads_started":1}
I20260806 23:54:42.061921 31867 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34845
I20260806 23:54:42.062228 31867 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:42.062788 31867 heartbeater.cc:511] Master 127.10.225.190:34845 requested a full tablet report, sending...
I20260806 23:54:42.063957 31390 ts_manager.cc:194] Registered new tserver with Master: 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691)
I20260806 23:54:42.064554 31390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:33877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:54:42.078712 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34845
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:42.199532 31874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:42.199968 31874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:42.200037 31874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:42.204401 31874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:42.204473 31874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:42.204536 31874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:54:42.210176 31874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:42.211355 31874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:42.212559 31874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:42.223582 31880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:42.223578 31881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:42.223624 31883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:42.223829 31874 server_base.cc:1034] running on GCE node
I20260806 23:54:42.224948 31874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:42.226052 31874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:42.227259 31874 hybrid_clock.cc:648] HybridClock initialized: now 1786060482227217 us; error 62 us; skew 500 ppm
Aug 06 23:54:42 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:42.231189 31874 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:54:42.232301 31874 webserver.cc:460] Webserver started at http://127.10.225.131:44545/ using document root <none> and password file <none>
I20260806 23:54:42.232896 31874 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:42.232971 31874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:42.233172 31874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:54:42.235355 31874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "20f9fba3083b46d3b6dd81813084bcef"
format_stamp: "Formatted at 2026-08-06 23:54:42 on dist-test-slave-nj21"
server_key: "8cedc420f6fd7cd58e13047701630ade"
server_key_iv: "cbeea9282cadfbca7940e752a1fd8bf8"
server_key_version: "encryptionkey@0"
I20260806 23:54:42.235972 31874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "20f9fba3083b46d3b6dd81813084bcef"
format_stamp: "Formatted at 2026-08-06 23:54:42 on dist-test-slave-nj21"
server_key: "8cedc420f6fd7cd58e13047701630ade"
server_key_iv: "cbeea9282cadfbca7940e752a1fd8bf8"
server_key_version: "encryptionkey@0"
I20260806 23:54:42.239939 31874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:54:42.242578 31892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:42.244405 31874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:42.244534 31874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "20f9fba3083b46d3b6dd81813084bcef"
format_stamp: "Formatted at 2026-08-06 23:54:42 on dist-test-slave-nj21"
server_key: "8cedc420f6fd7cd58e13047701630ade"
server_key_iv: "cbeea9282cadfbca7940e752a1fd8bf8"
server_key_version: "encryptionkey@0"
I20260806 23:54:42.244638 31874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:42.287897 31874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:42.291013 31874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:42.291348 31874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:42.292073 31874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:42.292992 31874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:42.293040 31874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:42.293097 31874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:42.293121 31874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:42.302038 31874 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:46501
I20260806 23:54:42.302062 32052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:46501 every 8 connection(s)
I20260806 23:54:42.302959 31874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:54:42.307834 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 31874
I20260806 23:54:42.308102 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:54:42.308600 11142 external_mini_cluster.cc:1428] Setting key a6c7ee0adcd756ffa4392e5d2b4920f4
Aug 06 23:54:42 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:42.318049 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.304882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:37923 (local address 127.10.225.190:34845)
0806 23:54:42.305072 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:42.305078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:42.305965 (+ 887us) server_negotiation.cc:407] Connection header received
0806 23:54:42.307402 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:42.307411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:42.307600 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:42.307840 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:42.309364 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.309983 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.311117 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.311334 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.311429 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.314015 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:42.314018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:42.314021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:42.314047 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:42.315423 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:42.316175 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:42.316178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:42.316180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:42.316266 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:42.316627 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:42.316630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:42.316632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:42.316965 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:42.317087 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:42.317708 (+ 621us) server_negotiation.cc:299] Negotiation successful
0806 23:54:42.317863 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 23:54:42.318917 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.305297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:37923)
0806 23:54:42.305801 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:42.305846 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:42.307059 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:42.308075 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:42.308082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:42.308545 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:42.309161 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.309182 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.310113 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.310117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.310997 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.311008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.311480 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.311484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.311649 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.312289 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:42.312310 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:42.313847 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:42.315623 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:42.315632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:42.315654 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:42.316063 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:42.316360 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:42.316363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:42.316365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:42.316533 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:42.317131 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:42.317139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:42.317427 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 23:54:42.317643 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 23:54:42.318019 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260806 23:54:42.320210 32053 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34845
I20260806 23:54:42.320487 32053 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:42.321019 32053 heartbeater.cc:511] Master 127.10.225.190:34845 requested a full tablet report, sending...
I20260806 23:54:42.322064 31390 ts_manager.cc:194] Registered new tserver with Master: 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501)
I20260806 23:54:42.322664 31390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:37923
I20260806 23:54:42.322993 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:54:42.332698 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.325541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.10.225.190:34845)
0806 23:54:42.325697 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:42.325702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:42.325830 (+ 128us) server_negotiation.cc:407] Connection header received
0806 23:54:42.325981 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:42.325984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:42.326027 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:42.326090 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:42.326797 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.327298 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.328173 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.328395 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.328479 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.329814 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:42.329817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:42.329819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:42.329842 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:42.331071 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:42.331474 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:42.331478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:42.331479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:42.331551 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:42.331853 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:42.331857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:42.331858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:42.332080 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:42.332201 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:42.332433 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 23:54:42.332540 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 23:54:42.335623 31390 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:54:42.338817 31390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:54:42.355525 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.348227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:45691 (local address 127.0.0.1:56724)
0806 23:54:42.348694 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:42.348740 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:42.348934 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:42.350647 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:42.350656 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:42.350692 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:42.350891 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.350900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.352692 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.352696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.353869 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.353879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.354968 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.354971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.355087 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.355136 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 23:54:42.355227 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 23:54:42.355361 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":28,"spinlock_wait_cycles":3072,"thread_start_us":190,"threads_started":1}
I20260806 23:54:42.356065 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.349862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.10.225.131:46501)
0806 23:54:42.350193 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:42.350204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:42.350226 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:54:42.350298 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:42.350303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:42.350486 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:42.350770 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:42.351162 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.352340 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.354148 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.354982 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.355377 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.355511 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:42.355556 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 23:54:42.355926 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":136,"threads_started":1}
I20260806 23:54:42.356076 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.349088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:46501 (local address 127.0.0.1:53480)
0806 23:54:42.349559 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:42.349574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:42.349698 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:42.350646 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:42.350651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:42.350689 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:42.351059 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.351065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.352485 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.352489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.354034 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.354038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.355112 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.355116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.355201 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.355227 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 23:54:42.355269 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 23:54:42.355948 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":862,"spinlock_wait_cycles":7680,"thread_start_us":99,"threads_started":1}
I20260806 23:54:42.356772 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.348459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.10.225.130:45691)
0806 23:54:42.350165 (+ 1706us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:42.350178 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:42.350201 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:54:42.350281 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:42.350286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:42.350496 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:42.351214 (+ 718us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:42.351381 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.352543 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.353988 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.354827 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.356340 (+ 1513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.356469 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:42.356517 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 23:54:42.356612 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1599,"thread_start_us":126,"threads_started":1}
I20260806 23:54:42.356822 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.349166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:43375 (local address 127.0.0.1:42950)
0806 23:54:42.349817 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:42.349831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:42.349920 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:42.351513 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:42.351516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:42.351534 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:42.351620 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.351626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.353751 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.353754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.354859 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.354869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.355925 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.355928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.356577 (+ 649us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.356600 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:54:42.356650 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 23:54:42.356714 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":111,"threads_started":1}
I20260806 23:54:42.357400 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.349488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.10.225.129:43375)
0806 23:54:42.350944 (+ 1456us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:42.350950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:42.350972 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:54:42.351048 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:42.351053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:42.351236 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:42.351380 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:42.351783 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.353634 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.354999 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.355793 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.356993 (+ 1200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.357119 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:42.357165 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 23:54:42.357251 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1355,"thread_start_us":134,"threads_started":1}
I20260806 23:54:42.359388 31750 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd8f68a2ba845aaacaea60aef075892 (DEFAULT_TABLE table=test-table [id=0b1143f1c11e4551ae2eed70e5a70a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:54:42.359391 31962 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd8f68a2ba845aaacaea60aef075892 (DEFAULT_TABLE table=test-table [id=0b1143f1c11e4551ae2eed70e5a70a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:54:42.359855 31565 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd8f68a2ba845aaacaea60aef075892 (DEFAULT_TABLE table=test-table [id=0b1143f1c11e4551ae2eed70e5a70a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:54:42.360459 31750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd8f68a2ba845aaacaea60aef075892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:42.360474 31962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd8f68a2ba845aaacaea60aef075892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:42.360765 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd8f68a2ba845aaacaea60aef075892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:42.366921 32081 tablet_bootstrap.cc:492] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: Bootstrap starting.
I20260806 23:54:42.368059 32082 tablet_bootstrap.cc:492] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Bootstrap starting.
I20260806 23:54:42.369961 32082 tablet_bootstrap.cc:654] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:42.370025 32081 tablet_bootstrap.cc:654] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:42.370688 32082 log.cc:824] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:42.370882 32081 log.cc:824] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:42.371392 32084 tablet_bootstrap.cc:492] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: Bootstrap starting.
I20260806 23:54:42.372658 32082 tablet_bootstrap.cc:492] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: No bootstrap required, opened a new log
I20260806 23:54:42.372857 32082 ts_tablet_manager.cc:1397] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 23:54:42.372856 32081 tablet_bootstrap.cc:492] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: No bootstrap required, opened a new log
I20260806 23:54:42.373034 32081 ts_tablet_manager.cc:1397] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:54:42.373536 32084 tablet_bootstrap.cc:654] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:42.374816 32084 log.cc:824] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:42.376933 32084 tablet_bootstrap.cc:492] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: No bootstrap required, opened a new log
I20260806 23:54:42.376839 32082 raft_consensus.cc:348] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:42.377080 32082 raft_consensus.cc:374] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:42.377137 32084 ts_tablet_manager.cc:1397] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:54:42.377143 32082 raft_consensus.cc:729] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99315324fc26442398b7cc25f29dafdc, State: Initialized, Role: FOLLOWER
I20260806 23:54:42.376933 32081 raft_consensus.cc:348] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:42.377229 32081 raft_consensus.cc:374] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:42.377285 32081 raft_consensus.cc:729] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83052a98317a496ab42ba3b97f224ca9, State: Initialized, Role: FOLLOWER
I20260806 23:54:42.377645 32082 consensus_queue.cc:260] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:42.377712 32081 consensus_queue.cc:260] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:42.378294 31660 heartbeater.cc:503] Master 127.10.225.190:34845 was elected leader, sending a full tablet report...
I20260806 23:54:42.378684 32081 ts_tablet_manager.cc:1428] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 23:54:42.379033 31867 heartbeater.cc:503] Master 127.10.225.190:34845 was elected leader, sending a full tablet report...
I20260806 23:54:42.379690 32082 ts_tablet_manager.cc:1428] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 23:54:42.381351 32084 raft_consensus.cc:348] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:42.381618 32084 raft_consensus.cc:374] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:42.381678 32084 raft_consensus.cc:729] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f9fba3083b46d3b6dd81813084bcef, State: Initialized, Role: FOLLOWER
I20260806 23:54:42.382174 32084 consensus_queue.cc:260] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:42.382826 32053 heartbeater.cc:503] Master 127.10.225.190:34845 was elected leader, sending a full tablet report...
I20260806 23:54:42.383242 32084 ts_tablet_manager.cc:1428] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260806 23:54:42.542649 31868 tablet.cc:2369] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:54:42.543856 31661 tablet.cc:2369] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:54:42.548643 32096 raft_consensus.cc:482] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:54:42.548861 32096 raft_consensus.cc:504] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:42.550155 32096 leader_election.cc:288] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691), 99315324fc26442398b7cc25f29dafdc (127.10.225.129:43375)
W20260806 23:54:42.553953 32054 tablet.cc:2369] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:54:42.554457 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.550481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:43375 (local address 127.10.225.131:35967)
0806 23:54:42.550605 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:42.550621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:42.550718 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:42.551063 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:42.551066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:42.551085 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:42.551201 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.551207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.552114 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.552118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.553212 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.553223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.553982 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.553985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.554095 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.554130 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 23:54:42.554212 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 23:54:42.554306 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 23:54:42.554490 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.550674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:35967 (local address 127.10.225.129:43375)
0806 23:54:42.550861 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:42.550865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:42.550879 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:54:42.550928 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:42.550931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:42.550970 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:42.551035 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:42.551357 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.551973 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.553367 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.553844 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.553986 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.554048 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:42.554244 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 23:54:42.554345 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 23:54:42.555190 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.550640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:45691 (local address 127.10.225.131:52385)
0806 23:54:42.551236 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:42.551255 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:42.551400 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:42.551827 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:42.551830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:42.551848 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:42.551988 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.551994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.552756 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.552759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.553751 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:42.553761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.554681 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.554686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:42.554815 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.554847 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 23:54:42.554930 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 23:54:42.555029 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":13,"thread_start_us":179,"threads_started":1}
I20260806 23:54:42.555210 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.551378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:52385 (local address 127.10.225.130:45691)
0806 23:54:42.551569 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:42.551573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:42.551587 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:54:42.551646 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:42.551648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:42.551731 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:42.551807 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:42.552139 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.552637 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.553907 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.554403 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.554524 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.554577 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:42.554981 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 23:54:42.555082 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 23:54:42.555548 31591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd8f68a2ba845aaacaea60aef075892" candidate_uuid: "20f9fba3083b46d3b6dd81813084bcef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99315324fc26442398b7cc25f29dafdc" is_pre_election: true
I20260806 23:54:42.555979 31591 raft_consensus.cc:2454] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f9fba3083b46d3b6dd81813084bcef in term 0.
I20260806 23:54:42.556113 31786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd8f68a2ba845aaacaea60aef075892" candidate_uuid: "20f9fba3083b46d3b6dd81813084bcef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83052a98317a496ab42ba3b97f224ca9" is_pre_election: true
I20260806 23:54:42.556504 31786 raft_consensus.cc:2454] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f9fba3083b46d3b6dd81813084bcef in term 0.
I20260806 23:54:42.556672 31899 leader_election.cc:302] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f9fba3083b46d3b6dd81813084bcef, 99315324fc26442398b7cc25f29dafdc; no voters:
I20260806 23:54:42.557044 32096 raft_consensus.cc:2781] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:54:42.557252 32096 raft_consensus.cc:482] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:54:42.557351 32096 raft_consensus.cc:3037] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:42.558491 32096 raft_consensus.cc:504] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:42.558929 32096 leader_election.cc:288] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [CANDIDATE]: Term 1 election: Requested vote from peers 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691), 99315324fc26442398b7cc25f29dafdc (127.10.225.129:43375)
I20260806 23:54:42.559262 31786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd8f68a2ba845aaacaea60aef075892" candidate_uuid: "20f9fba3083b46d3b6dd81813084bcef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83052a98317a496ab42ba3b97f224ca9"
I20260806 23:54:42.559388 31786 raft_consensus.cc:3037] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:42.559454 31591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd8f68a2ba845aaacaea60aef075892" candidate_uuid: "20f9fba3083b46d3b6dd81813084bcef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99315324fc26442398b7cc25f29dafdc"
I20260806 23:54:42.559639 31591 raft_consensus.cc:3037] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:42.560565 31786 raft_consensus.cc:2454] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f9fba3083b46d3b6dd81813084bcef in term 1.
I20260806 23:54:42.560864 31893 leader_election.cc:302] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f9fba3083b46d3b6dd81813084bcef, 83052a98317a496ab42ba3b97f224ca9; no voters:
I20260806 23:54:42.560988 31591 raft_consensus.cc:2454] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f9fba3083b46d3b6dd81813084bcef in term 1.
I20260806 23:54:42.561026 32096 raft_consensus.cc:2781] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:54:42.561261 32096 raft_consensus.cc:686] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 1 LEADER]: Becoming Leader. State: Replica: 20f9fba3083b46d3b6dd81813084bcef, State: Running, Role: LEADER
I20260806 23:54:42.561650 32096 consensus_queue.cc:237] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:42.564877 31390 catalog_manager.cc:5495] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131). New cstate: current_term: 1 leader_uuid: "20f9fba3083b46d3b6dd81813084bcef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } health_report { overall_health: UNKNOWN } } }
I20260806 23:54:42.598412 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.594379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.10.225.131:46501)
0806 23:54:42.594553 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:42.594558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:42.594599 (+ 41us) server_negotiation.cc:407] Connection header received
0806 23:54:42.594711 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:42.594714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:42.594791 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:42.594894 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:54:42.595341 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.596030 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.596902 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:42.597120 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:42.597209 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:42.597533 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:54:42.598006 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:54:42.598108 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:42.598191 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 23:54:42.598252 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 23:54:42.605067 31591 raft_consensus.cc:1261] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Refusing update from remote peer 20f9fba3083b46d3b6dd81813084bcef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:54:42.605067 31786 raft_consensus.cc:1261] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [term 1 FOLLOWER]: Refusing update from remote peer 20f9fba3083b46d3b6dd81813084bcef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:54:42.605830 32096 consensus_queue.cc:1035] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:54:42.606098 32100 consensus_queue.cc:1035] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:54:42.615404 32105 mvcc.cc:205] Tried to move back new op lower bound from 7315703736742309888 to 7315703736575193088. Current Snapshot: MvccSnapshot[applied={T|T < 7315703736742309888}]
I20260806 23:54:42.624017 31388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:54:42.624228 31388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:54:42.626931 31388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Sending DeleteTablet for 3 replicas of tablet 5bd8f68a2ba845aaacaea60aef075892
I20260806 23:54:42.627882 31962 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd8f68a2ba845aaacaea60aef075892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:54:42 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:53480
I20260806 23:54:42.627864 31750 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd8f68a2ba845aaacaea60aef075892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:54:42 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:56724
I20260806 23:54:42.627874 31565 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd8f68a2ba845aaacaea60aef075892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:54:42 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:42950
I20260806 23:54:42.628094 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 31479
I20260806 23:54:42.641738 32119 tablet_replica.cc:330] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: stopping tablet replica
I20260806 23:54:42.642172 32119 raft_consensus.cc:2229] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:54:42.642524 32119 raft_consensus.cc:2258] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:54:42.643159 32120 tablet_replica.cc:330] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: stopping tablet replica
I20260806 23:54:42.643452 32120 raft_consensus.cc:2229] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:54:42.644212 32120 raft_consensus.cc:2258] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:54:42.646003 32119 ts_tablet_manager.cc:1905] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:54:42.648840 32119 ts_tablet_manager.cc:1918] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:54:42.649022 32119 log.cc:1196] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/5bd8f68a2ba845aaacaea60aef075892
I20260806 23:54:42.649641 32119 ts_tablet_manager.cc:1939] T 5bd8f68a2ba845aaacaea60aef075892 P 83052a98317a496ab42ba3b97f224ca9: Deleting consensus metadata
I20260806 23:54:42.650115 32120 ts_tablet_manager.cc:1905] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:54:42.651283 31370 catalog_manager.cc:4841] TS 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691): tablet 5bd8f68a2ba845aaacaea60aef075892 (table test-table [id=0b1143f1c11e4551ae2eed70e5a70a91]) successfully deleted
I20260806 23:54:42.652894 32120 ts_tablet_manager.cc:1918] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:54:42.653038 32120 log.cc:1196] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/5bd8f68a2ba845aaacaea60aef075892
W20260806 23:54:42.653309 31376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:43375 (error 108)
I20260806 23:54:42.653383 32120 ts_tablet_manager.cc:1939] T 5bd8f68a2ba845aaacaea60aef075892 P 20f9fba3083b46d3b6dd81813084bcef: Deleting consensus metadata
I20260806 23:54:42.653932 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 31667
I20260806 23:54:42.676344 31370 catalog_manager.cc:4841] TS 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501): tablet 5bd8f68a2ba845aaacaea60aef075892 (table test-table [id=0b1143f1c11e4551ae2eed70e5a70a91]) successfully deleted
I20260806 23:54:42.677078 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:42.676564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:43375 (local address 127.0.0.1:42964)
0806 23:54:42.676844 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:42.676962 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260806 23:54:42.677327 31376 catalog_manager.cc:4584] TS 99315324fc26442398b7cc25f29dafdc (127.10.225.129:43375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bd8f68a2ba845aaacaea60aef075892: Network error: Client connection negotiation failed: client connection to 127.10.225.129:43375: connect: Connection refused (error 111)
I20260806 23:54:42.684420 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 31874
I20260806 23:54:42.696517 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 31349
I20260806 23:54:42.707652 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34845
--webserver_interface=127.10.225.190
--webserver_port=34315
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:42.830739 32122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:42.831036 32122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:42.831079 32122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:42.835428 32122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:54:42.835526 32122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:42.835553 32122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:42.835577 32122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:54:42.835609 32122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:54:42.841276 32122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=34315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:42.842434 32122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:42.843505 32122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:42.857622 32131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:42.857720 32134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:42.857801 32122 server_base.cc:1034] running on GCE node
W20260806 23:54:42.857724 32130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:42.858659 32122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:42.859941 32122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:42.861133 32122 hybrid_clock.cc:648] HybridClock initialized: now 1786060482861094 us; error 52 us; skew 500 ppm
Aug 06 23:54:42 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:42.864244 32122 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:54:42.865269 32122 webserver.cc:460] Webserver started at http://127.10.225.190:34315/ using document root <none> and password file <none>
I20260806 23:54:42.865875 32122 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:42.865959 32122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:42.870546 32122 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:54:42.872937 32145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:42.874289 32122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:54:42.874421 32122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "1c0e22cc086c45d7a023d019fb695d8e"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "759821544b1c1e530156853b737236c5"
server_key_iv: "332c0a714b645d938c0338e8d617b426"
server_key_version: "encryptionkey@0"
I20260806 23:54:42.875084 32122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:42.884130 32122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:42.887346 32122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:42.887516 32122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:42.896088 32122 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:34845
I20260806 23:54:42.896369 32220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:34845 every 8 connection(s)
I20260806 23:54:42.897543 32122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:54:42.903144 32223 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:54:42.905184 32223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Bootstrap starting.
I20260806 23:54:42.905520 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 32122
I20260806 23:54:42.905995 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:43375
--local_ip_for_outbound_sockets=127.10.225.129
--tserver_master_addrs=127.10.225.190:34845
--webserver_port=39727
--webserver_interface=127.10.225.129
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:54:42.929675 32223 log.cc:824] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:42.941188 32223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:54:42.941665 32223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Bootstrap complete.
I20260806 23:54:42.947328 32223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:42.948004 32223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e22cc086c45d7a023d019fb695d8e, State: Initialized, Role: FOLLOWER
I20260806 23:54:42.948549 32223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:42.948668 32223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:54:42.948740 32223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:54:42.948836 32223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:54:42.950904 32223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:42.951247 32223 leader_election.cc:302] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c0e22cc086c45d7a023d019fb695d8e; no voters:
I20260806 23:54:42.951566 32223 leader_election.cc:288] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 23:54:42.951721 32229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 23:54:42.951979 32229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 2 LEADER]: Becoming Leader. State: Replica: 1c0e22cc086c45d7a023d019fb695d8e, State: Running, Role: LEADER
I20260806 23:54:42.952306 32229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:42.952844 32223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:54:42.953815 32230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c0e22cc086c45d7a023d019fb695d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } } }
I20260806 23:54:42.953945 32230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:42.954300 32244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:54:42.954293 32232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0e22cc086c45d7a023d019fb695d8e. Latest consensus state: current_term: 2 leader_uuid: "1c0e22cc086c45d7a023d019fb695d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } } }
I20260806 23:54:42.954398 32232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:42.959405 32244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:54:42.960114 32244 catalog_manager.cc:1259] Loaded cluster ID: 8874634a975d4d95bafe3b1aa40827fb
I20260806 23:54:42.960176 32244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:54:42.961637 32244 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 23:54:42.966197 32244 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 23:54:42.971947 32244 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:54:42.972779 32244 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Loaded TSK: 0
I20260806 23:54:42.974026 32244 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 23:54:43.044217 32226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:43.044479 32226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:43.044525 32226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:43.048574 32226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:43.048637 32226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:43.048700 32226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:54:43.054193 32226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:43375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=39727
--tserver_master_addrs=127.10.225.190:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:43.055260 32226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:43.056324 32226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:43.066397 32267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:43.066401 32268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:43.066754 32270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:43.067082 32226 server_base.cc:1034] running on GCE node
I20260806 23:54:43.067615 32226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:43.068482 32226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:43.069691 32226 hybrid_clock.cc:648] HybridClock initialized: now 1786060483069667 us; error 35 us; skew 500 ppm
Aug 06 23:54:43 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060483, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:43.076402 32226 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:54:43.077831 32226 webserver.cc:460] Webserver started at http://127.10.225.129:39727/ using document root <none> and password file <none>
I20260806 23:54:43.078709 32226 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:43.078840 32226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:43.084484 32226 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:54:43.087301 32280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:43.088903 32226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:43.089068 32226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "99315324fc26442398b7cc25f29dafdc"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "47075a54a51e19ee713c4b3f2b093366"
server_key_iv: "6ef231a747360e4906a6b2bbbddf5d70"
server_key_version: "encryptionkey@0"
I20260806 23:54:43.089847 32226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:43.124205 32226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:43.127508 32226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:43.127729 32226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:43.128413 32226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:43.129799 32290 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 23:54:43.134303 32226 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 23:54:43.134392 32226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260806 23:54:43.134470 32226 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 23:54:43.136336 32226 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 23:54:43.136385 32226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260806 23:54:43.136737 32290 tablet_bootstrap.cc:492] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Bootstrap starting.
I20260806 23:54:43.146777 32226 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:43375
I20260806 23:54:43.146833 32449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:43375 every 8 connection(s)
I20260806 23:54:43.148154 32226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:54:43.148734 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 32226
I20260806 23:54:43.149225 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:45691
--local_ip_for_outbound_sockets=127.10.225.130
--tserver_master_addrs=127.10.225.190:34845
--webserver_port=39225
--webserver_interface=127.10.225.130
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:54:43.157053 32290 log.cc:824] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:54:43 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060483, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:43.165452 32290 tablet_bootstrap.cc:492] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:54:43.165870 32290 tablet_bootstrap.cc:492] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Bootstrap complete.
I20260806 23:54:43.167181 32290 ts_tablet_manager.cc:1397] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.013s
I20260806 23:54:43.169278 32457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.150921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:51169)
0806 23:54:43.151828 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:43.151907 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:43.153188 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:43.154530 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:43.154544 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:43.155072 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:43.155745 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.155769 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.157843 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.157847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.159233 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.159247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.160293 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.160298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.160514 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.161249 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:43.161270 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:43.163339 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:43.165831 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:43.165837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:43.165850 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:43.166207 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:43.166594 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:43.166597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:43.166599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:43.166766 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:43.167448 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:43.167455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:43.167763 (+ 308us) client_negotiation.cc:769] Sending connection context
0806 23:54:43.167963 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 23:54:43.168303 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":159,"threads_started":1}
I20260806 23:54:43.169276 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.150840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:51169 (local address 127.10.225.190:34845)
0806 23:54:43.151414 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.151436 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.152065 (+ 629us) server_negotiation.cc:407] Connection header received
0806 23:54:43.153718 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.153728 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.154033 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.154395 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:43.155946 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.157630 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.159388 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.160112 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.160292 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.163577 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:43.163585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:43.163604 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:43.163642 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:43.165688 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.166325 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.166339 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.166345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.166499 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.166853 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.166856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.166859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.167310 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:43.167453 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.168082 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.168396 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":145,"threads_started":1}
I20260806 23:54:43.170225 32290 raft_consensus.cc:348] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.170755 32290 raft_consensus.cc:729] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99315324fc26442398b7cc25f29dafdc, State: Initialized, Role: FOLLOWER
I20260806 23:54:43.171223 32290 consensus_queue.cc:260] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.172344 32450 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34845
I20260806 23:54:43.172564 32290 ts_tablet_manager.cc:1428] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:54:43.172578 32450 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:43.173164 32450 heartbeater.cc:511] Master 127.10.225.190:34845 requested a full tablet report, sending...
I20260806 23:54:43.175896 32169 ts_manager.cc:194] Registered new tserver with Master: 99315324fc26442398b7cc25f29dafdc (127.10.225.129:43375)
I20260806 23:54:43.180002 32169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:51169
I20260806 23:54:43.182750 32450 heartbeater.cc:503] Master 127.10.225.190:34845 was elected leader, sending a full tablet report...
Aug 06 23:54:43 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060482, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:54:43.191059 32465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.179772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.10.225.129:43375)
0806 23:54:43.180792 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.180803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.180826 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:54:43.180899 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.180903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.181059 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.181174 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:43.182150 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.183116 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.184140 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.184549 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.184644 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.186992 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:43.187004 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:43.187020 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:43.187058 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:43.188997 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.189602 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.189608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.189609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.189702 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.189988 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.189990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.189991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.190243 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:43.190344 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.190747 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.190906 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":101,"threads_started":1}
I20260806 23:54:43.191097 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.179642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:43375 (local address 127.0.0.1:42974)
0806 23:54:43.180426 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:43.180461 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:43.180578 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:43.181198 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:43.181203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:43.181774 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:43.181963 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.181974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.183241 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.183246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.184008 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.184020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.184659 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.184663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.184771 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.185295 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:43.185307 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:43.186794 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:43.189112 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:43.189122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:43.189128 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:43.189450 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:43.189792 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:43.189796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:43.189798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:43.189904 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:43.190371 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:43.190378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:43.190533 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 23:54:43.190691 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 23:54:43.190918 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":131,"threads_started":1}
I20260806 23:54:43.192194 32350 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd8f68a2ba845aaacaea60aef075892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:54:42 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:42974
I20260806 23:54:43.193053 32466 tablet_replica.cc:330] stopping tablet replica
I20260806 23:54:43.193343 32466 raft_consensus.cc:2229] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:54:43.193594 32466 raft_consensus.cc:2258] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:54:43.194561 32466 ts_tablet_manager.cc:1905] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:54:43.198756 32466 ts_tablet_manager.cc:1918] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:54:43.198858 32466 log.cc:1196] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/5bd8f68a2ba845aaacaea60aef075892
I20260806 23:54:43.199102 32466 ts_tablet_manager.cc:1939] T 5bd8f68a2ba845aaacaea60aef075892 P 99315324fc26442398b7cc25f29dafdc: Deleting consensus metadata
I20260806 23:54:43.200929 32152 catalog_manager.cc:4841] TS 99315324fc26442398b7cc25f29dafdc (127.10.225.129:43375): tablet 5bd8f68a2ba845aaacaea60aef075892 (table test-table [id=0b1143f1c11e4551ae2eed70e5a70a91]) successfully deleted
W20260806 23:54:43.287225 32459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:43.287540 32459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:43.287607 32459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:43.291813 32459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:43.291893 32459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:43.291982 32459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:54:43.297637 32459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:45691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=39225
--tserver_master_addrs=127.10.225.190:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:43.298790 32459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:43.299887 32459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:43.310283 32472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:43.310366 32459 server_base.cc:1034] running on GCE node
W20260806 23:54:43.310281 32474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:43.310281 32477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:43.311031 32459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:43.311730 32459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:43.312917 32459 hybrid_clock.cc:648] HybridClock initialized: now 1786060483312879 us; error 47 us; skew 500 ppm
Aug 06 23:54:43 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060483, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:43.316370 32459 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:54:43.317274 32459 webserver.cc:460] Webserver started at http://127.10.225.130:39225/ using document root <none> and password file <none>
I20260806 23:54:43.317828 32459 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:43.317981 32459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:43.321975 32459 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:54:43.324190 32485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:43.326045 32459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:43.326164 32459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "83052a98317a496ab42ba3b97f224ca9"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "a932a9ce2f9b850e0541519e3c0053f9"
server_key_iv: "2526c2894d066463fc56f01f25013256"
server_key_version: "encryptionkey@0"
I20260806 23:54:43.326764 32459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:43.362421 32459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:43.365682 32459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:43.365854 32459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:43.367122 32459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:43.368060 32459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:43.368111 32459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:43.368155 32459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:43.368175 32459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:43.378127 32459 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:45691
I20260806 23:54:43.378172 32646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:45691 every 8 connection(s)
I20260806 23:54:43.379281 32459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:54:43.380882 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 32459
I20260806 23:54:43.381529 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:46501
--local_ip_for_outbound_sockets=127.10.225.131
--tserver_master_addrs=127.10.225.190:34845
--webserver_port=44545
--webserver_interface=127.10.225.131
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:54:43 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060483, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:43.396584 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.381324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:45845 (local address 127.10.225.190:34845)
0806 23:54:43.381505 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.381510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.382670 (+ 1160us) server_negotiation.cc:407] Connection header received
0806 23:54:43.384559 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.384624 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.384720 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.384845 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:43.386224 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.386757 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.388336 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.388571 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.388660 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.392029 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:43.392033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:43.392036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:43.392068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:43.393988 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.394548 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.394552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.394554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.394647 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.394955 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.394958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.394960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.395193 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:43.395293 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.396244 (+ 951us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.396402 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 23:54:43.397893 32650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.381779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:45845)
0806 23:54:43.382368 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:43.382486 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:43.384265 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:43.385077 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:43.385085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:43.385494 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:43.386063 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.386082 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.386894 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.386897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.388213 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.388225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.388719 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.388725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.388900 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.389637 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:43.389660 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:43.391823 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:43.394138 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:43.394144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:43.394159 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:43.394447 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:43.394727 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:43.394729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:43.394730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:43.394870 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:43.395362 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:43.395370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:43.395859 (+ 489us) client_negotiation.cc:769] Sending connection context
0806 23:54:43.396159 (+ 300us) client_negotiation.cc:240] Negotiation successful
0806 23:54:43.396663 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":119,"threads_started":1}
I20260806 23:54:43.399482 32647 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34845
I20260806 23:54:43.399813 32647 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:43.400388 32647 heartbeater.cc:511] Master 127.10.225.190:34845 requested a full tablet report, sending...
I20260806 23:54:43.401713 32169 ts_manager.cc:194] Registered new tserver with Master: 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691)
I20260806 23:54:43.402341 32169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:45845
W20260806 23:54:43.512146 32651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:43.512454 32651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:43.512503 32651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:43.516776 32651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:43.516850 32651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:43.516925 32651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:54:43.522517 32651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:46501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=44545
--tserver_master_addrs=127.10.225.190:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:43.523720 32651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:43.524824 32651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:43.535378 32657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:43.535429 32658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:43.535441 32661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:43.535648 32651 server_base.cc:1034] running on GCE node
I20260806 23:54:43.536137 32651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:43.536818 32651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:43.538012 32651 hybrid_clock.cc:648] HybridClock initialized: now 1786060483537985 us; error 39 us; skew 500 ppm
Aug 06 23:54:43 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060483, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:43.540990 32651 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:54:43.541945 32651 webserver.cc:460] Webserver started at http://127.10.225.131:44545/ using document root <none> and password file <none>
I20260806 23:54:43.542505 32651 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:43.542582 32651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:43.546702 32651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:54:43.549008 32675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:43.550482 32651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:54:43.550613 32651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "20f9fba3083b46d3b6dd81813084bcef"
format_stamp: "Formatted at 2026-08-06 23:54:42 on dist-test-slave-nj21"
server_key: "8cedc420f6fd7cd58e13047701630ade"
server_key_iv: "cbeea9282cadfbca7940e752a1fd8bf8"
server_key_version: "encryptionkey@0"
I20260806 23:54:43.551193 32651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:43.565397 32651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:43.568576 32651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:43.568769 32651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:43.569401 32651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:43.570254 32651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:43.570302 32651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:43.570346 32651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:43.570374 32651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:43.580693 32651 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:46501
I20260806 23:54:43.580714 373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:46501 every 8 connection(s)
I20260806 23:54:43.581864 32651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:54:43.589862 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 32651
Aug 06 23:54:43 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060483, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:54:43.597867 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.583930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:47843 (local address 127.10.225.190:34845)
0806 23:54:43.584168 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.584174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.584969 (+ 795us) server_negotiation.cc:407] Connection header received
0806 23:54:43.586461 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.586464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.586519 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.586598 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:43.587946 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.588484 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.589769 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.590029 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.590119 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.592979 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:43.592984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:43.592987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:43.593025 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:43.595006 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.595719 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.595724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.595726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.595811 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.596196 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.596199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.596201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.596439 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:43.596684 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.597473 (+ 789us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.597650 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 23:54:43.598800 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.584274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:47843)
0806 23:54:43.584813 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:43.584859 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:43.586172 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:43.586844 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:43.586853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:43.587241 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:43.587798 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.587815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.588623 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.588626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.589614 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.589627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.590191 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.590195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.590375 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.590983 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:43.591006 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:43.592789 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:43.595153 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:43.595162 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:43.595184 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:43.595549 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:43.595902 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:43.595906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:43.595908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:43.596087 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:43.596747 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:43.596756 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:43.597145 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 23:54:43.597351 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 23:54:43.597761 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20260806 23:54:43.600183 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.590705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.10.225.190:34845)
0806 23:54:43.590992 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.590996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.591071 (+ 75us) server_negotiation.cc:407] Connection header received
0806 23:54:43.591324 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.591328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.591411 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.591540 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:43.592353 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.592992 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.594310 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.594669 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.594800 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.596311 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:43.596315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:43.596318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:43.596339 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:43.598229 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.598696 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.598699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.598701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.598776 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:43.599072 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:43.599076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:43.599078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:43.599309 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:43.599550 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.599854 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.600011 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
I20260806 23:54:43.600811 374 heartbeater.cc:347] Connected to a master server at 127.10.225.190:34845
I20260806 23:54:43.601132 374 heartbeater.cc:464] Registering TS with master...
I20260806 23:54:43.601886 374 heartbeater.cc:511] Master 127.10.225.190:34845 requested a full tablet report, sending...
I20260806 23:54:43.603205 32169 ts_manager.cc:194] Registered new tserver with Master: 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501)
I20260806 23:54:43.604014 32169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:47843
I20260806 23:54:43.614566 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:54:43.619127 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.615275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.10.225.190:34845)
0806 23:54:43.615441 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.615446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.615584 (+ 138us) server_negotiation.cc:407] Connection header received
0806 23:54:43.615799 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.615802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.615924 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.616020 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:54:43.616271 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.616811 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.617669 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.617891 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.617984 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.618294 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:54:43.618731 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:54:43.618839 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.618906 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.618972 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 23:54:43.619856 32169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:54:43.623174 32169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:54:43.631927 32350 tablet_service.cc:1467] Processing CreateTablet for tablet 20dd24e88ea44de8b2a4ce6907b87376 (DEFAULT_TABLE table=test-table [id=ef20830db83e4b098fbb4b93d1a11cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:54:43.632689 32350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dd24e88ea44de8b2a4ce6907b87376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:43.635797 32290 tablet_bootstrap.cc:492] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Bootstrap starting.
I20260806 23:54:43.636071 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.630828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:46501 (local address 127.0.0.1:53500)
0806 23:54:43.630973 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:43.630989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:43.631066 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:43.632277 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:43.632280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:43.632301 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:43.632419 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.632426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.633665 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.633669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.634728 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.634738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.635584 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.635588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.635714 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.635758 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 23:54:43.635854 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 23:54:43.635938 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 23:54:43.636121 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.631245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.10.225.131:46501)
0806 23:54:43.631742 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.631753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.631791 (+ 38us) server_negotiation.cc:407] Connection header received
0806 23:54:43.631871 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.631877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.632112 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.632592 (+ 480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:43.632744 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.633528 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.634878 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.635463 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.635583 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.635783 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.635888 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.635989 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":124,"threads_started":1}
I20260806 23:54:43.636763 32290 tablet_bootstrap.cc:654] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:43.636783 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.631050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:45691 (local address 127.0.0.1:56726)
0806 23:54:43.631379 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:43.631392 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:43.631494 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:43.632634 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:43.632637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:43.632657 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:43.632754 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.632760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.634203 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.634207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.635292 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.635304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.636389 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.636392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.636510 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.636534 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:54:43.636586 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 23:54:43.636654 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":147,"threads_started":1}
I20260806 23:54:43.637553 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.631646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.10.225.130:45691)
0806 23:54:43.632026 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.632036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.632056 (+ 20us) server_negotiation.cc:407] Connection header received
0806 23:54:43.632284 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.632293 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.632465 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.632600 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:43.632892 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.633927 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.635466 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.636247 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.637103 (+ 856us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.637241 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.637298 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.637410 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":144,"threads_started":1}
I20260806 23:54:43.638447 32290 tablet_bootstrap.cc:492] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: No bootstrap required, opened a new log
I20260806 23:54:43.638514 32290 ts_tablet_manager.cc:1397] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 23:54:43.638985 32290 raft_consensus.cc:348] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.639114 32290 raft_consensus.cc:374] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:43.639153 32290 raft_consensus.cc:729] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99315324fc26442398b7cc25f29dafdc, State: Initialized, Role: FOLLOWER
I20260806 23:54:43.639313 32290 consensus_queue.cc:260] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.639586 32290 ts_tablet_manager.cc:1428] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 23:54:43.640770 32548 tablet_service.cc:1467] Processing CreateTablet for tablet 20dd24e88ea44de8b2a4ce6907b87376 (DEFAULT_TABLE table=test-table [id=ef20830db83e4b098fbb4b93d1a11cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:54:43.640892 32741 tablet_service.cc:1467] Processing CreateTablet for tablet 20dd24e88ea44de8b2a4ce6907b87376 (DEFAULT_TABLE table=test-table [id=ef20830db83e4b098fbb4b93d1a11cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:54:43.642024 32548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dd24e88ea44de8b2a4ce6907b87376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:43.642042 32741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dd24e88ea44de8b2a4ce6907b87376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:54:43.649035 397 tablet_bootstrap.cc:492] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: Bootstrap starting.
I20260806 23:54:43.649930 396 tablet_bootstrap.cc:492] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: Bootstrap starting.
I20260806 23:54:43.651531 397 tablet_bootstrap.cc:654] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: Neither blocks nor log segments found. Creating new log.
I20260806 23:54:43.651774 396 tablet_bootstrap.cc:654] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: Neither blocks nor log segments found. Creating new log.
W20260806 23:54:43.652560 32455 tablet.cc:2369] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:54:43.652365 397 log.cc:824] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:43.653040 396 log.cc:824] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:43.654769 397 tablet_bootstrap.cc:492] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: No bootstrap required, opened a new log
I20260806 23:54:43.654932 396 tablet_bootstrap.cc:492] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: No bootstrap required, opened a new log
I20260806 23:54:43.654991 397 ts_tablet_manager.cc:1397] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:54:43.655117 396 ts_tablet_manager.cc:1397] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:54:43.658085 397 raft_consensus.cc:348] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.658318 397 raft_consensus.cc:374] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:43.658371 397 raft_consensus.cc:729] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83052a98317a496ab42ba3b97f224ca9, State: Initialized, Role: FOLLOWER
I20260806 23:54:43.658843 397 consensus_queue.cc:260] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.658802 396 raft_consensus.cc:348] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.659179 396 raft_consensus.cc:374] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:54:43.659250 396 raft_consensus.cc:729] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f9fba3083b46d3b6dd81813084bcef, State: Initialized, Role: FOLLOWER
I20260806 23:54:43.659657 396 consensus_queue.cc:260] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.659942 397 ts_tablet_manager.cc:1428] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 23:54:43.660120 32647 heartbeater.cc:503] Master 127.10.225.190:34845 was elected leader, sending a full tablet report...
I20260806 23:54:43.660471 374 heartbeater.cc:503] Master 127.10.225.190:34845 was elected leader, sending a full tablet report...
I20260806 23:54:43.660624 396 ts_tablet_manager.cc:1428] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
W20260806 23:54:43.832921 375 tablet.cc:2369] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:54:43.862669 403 raft_consensus.cc:482] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:54:43.862911 403 raft_consensus.cc:504] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.864329 403 leader_election.cc:288] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691), 99315324fc26442398b7cc25f29dafdc (127.10.225.129:43375)
I20260806 23:54:43.868856 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.864680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:43375 (local address 127.10.225.131:35873)
0806 23:54:43.864904 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:43.864924 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:43.865080 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:43.865601 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:43.865606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:43.865636 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:43.865810 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.865819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.866645 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.866649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.867610 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.867620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.868429 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.868433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.868540 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.868568 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 23:54:43.868642 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 23:54:43.868714 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 23:54:43.868901 407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.864943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:35873 (local address 127.10.225.129:43375)
0806 23:54:43.865224 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.865229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.865246 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:54:43.865328 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.865331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.865412 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.865536 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:43.865949 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.866531 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.867797 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.868315 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.868417 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.868550 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.868663 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.868740 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":107,"threads_started":1}
I20260806 23:54:43.869011 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.864943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:43457 (local address 127.10.225.130:45691)
0806 23:54:43.865070 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.865075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.865536 (+ 461us) server_negotiation.cc:407] Connection header received
0806 23:54:43.865653 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.865656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.865721 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.865835 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:54:43.866212 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.866826 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.868026 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.868430 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.868557 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.868608 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.868805 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.868885 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:54:43.869026 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.864839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:45691 (local address 127.10.225.131:43457)
0806 23:54:43.865356 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:43.865372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:43.865520 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:43.865853 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:43.865856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:43.865874 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:54:43.866024 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.866031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.866985 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.866988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.867906 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:43.867917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.868564 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.868568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:43.868678 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.868701 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:54:43.868774 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 23:54:43.868841 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":1,"thread_start_us":156,"threads_started":1}
I20260806 23:54:43.869585 32379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dd24e88ea44de8b2a4ce6907b87376" candidate_uuid: "20f9fba3083b46d3b6dd81813084bcef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99315324fc26442398b7cc25f29dafdc" is_pre_election: true
I20260806 23:54:43.869659 32576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dd24e88ea44de8b2a4ce6907b87376" candidate_uuid: "20f9fba3083b46d3b6dd81813084bcef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83052a98317a496ab42ba3b97f224ca9" is_pre_election: true
I20260806 23:54:43.869891 32379 raft_consensus.cc:2454] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f9fba3083b46d3b6dd81813084bcef in term 0.
I20260806 23:54:43.869940 32576 raft_consensus.cc:2454] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f9fba3083b46d3b6dd81813084bcef in term 0.
I20260806 23:54:43.870380 32676 leader_election.cc:302] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20f9fba3083b46d3b6dd81813084bcef, 83052a98317a496ab42ba3b97f224ca9, 99315324fc26442398b7cc25f29dafdc; no voters:
I20260806 23:54:43.870643 403 raft_consensus.cc:2781] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:54:43.870726 403 raft_consensus.cc:482] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:54:43.870775 403 raft_consensus.cc:3037] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:43.872418 403 raft_consensus.cc:504] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.872779 403 leader_election.cc:288] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [CANDIDATE]: Term 1 election: Requested vote from peers 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691), 99315324fc26442398b7cc25f29dafdc (127.10.225.129:43375)
I20260806 23:54:43.873142 32379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dd24e88ea44de8b2a4ce6907b87376" candidate_uuid: "20f9fba3083b46d3b6dd81813084bcef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99315324fc26442398b7cc25f29dafdc"
I20260806 23:54:43.873169 32576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dd24e88ea44de8b2a4ce6907b87376" candidate_uuid: "20f9fba3083b46d3b6dd81813084bcef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83052a98317a496ab42ba3b97f224ca9"
I20260806 23:54:43.873267 32379 raft_consensus.cc:3037] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:43.873292 32576 raft_consensus.cc:3037] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:54:43.874296 32379 raft_consensus.cc:2454] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f9fba3083b46d3b6dd81813084bcef in term 1.
I20260806 23:54:43.874456 32576 raft_consensus.cc:2454] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f9fba3083b46d3b6dd81813084bcef in term 1.
I20260806 23:54:43.874594 32680 leader_election.cc:302] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f9fba3083b46d3b6dd81813084bcef, 99315324fc26442398b7cc25f29dafdc; no voters:
I20260806 23:54:43.874759 403 raft_consensus.cc:2781] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:54:43.874975 403 raft_consensus.cc:686] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 1 LEADER]: Becoming Leader. State: Replica: 20f9fba3083b46d3b6dd81813084bcef, State: Running, Role: LEADER
I20260806 23:54:43.875311 403 consensus_queue.cc:237] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:43.878640 32169 catalog_manager.cc:5495] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131). New cstate: current_term: 1 leader_uuid: "20f9fba3083b46d3b6dd81813084bcef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } health_report { overall_health: UNKNOWN } } }
W20260806 23:54:43.880616 32648 tablet.cc:2369] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:54:43.938079 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:43.934247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.10.225.131:46501)
0806 23:54:43.934379 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:43.934383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:43.934397 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:54:43.934519 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:43.934522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:43.934579 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:43.934671 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:54:43.934946 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.935652 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.936556 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:43.936806 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:43.936890 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:43.937202 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:54:43.937704 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:54:43.937814 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:43.937882 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 23:54:43.937938 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:54:43.945621 32379 raft_consensus.cc:1261] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Refusing update from remote peer 20f9fba3083b46d3b6dd81813084bcef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:54:43.945631 32576 raft_consensus.cc:1261] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [term 1 FOLLOWER]: Refusing update from remote peer 20f9fba3083b46d3b6dd81813084bcef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:54:43.946352 403 consensus_queue.cc:1035] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:54:43.946609 408 consensus_queue.cc:1035] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:54:43.953461 412 mvcc.cc:205] Tried to move back new op lower bound from 7315703742231437312 to 7315703741956292608. Current Snapshot: MvccSnapshot[applied={T|T < 7315703742231437312}]
I20260806 23:54:43.954535 410 mvcc.cc:205] Tried to move back new op lower bound from 7315703742231437312 to 7315703741956292608. Current Snapshot: MvccSnapshot[applied={T|T < 7315703742231437312 or (T in {7315703742231437312})}]
I20260806 23:54:43.957954 411 mvcc.cc:205] Tried to move back new op lower bound from 7315703742231437312 to 7315703741956292608. Current Snapshot: MvccSnapshot[applied={T|T < 7315703742231437312 or (T in {7315703742231437312})}]
I20260806 23:54:43.962283 32169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:54:43.962482 32169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:54:43.965147 32169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Sending DeleteTablet for 3 replicas of tablet 20dd24e88ea44de8b2a4ce6907b87376
I20260806 23:54:43.966071 32350 tablet_service.cc:1514] Processing DeleteTablet for tablet 20dd24e88ea44de8b2a4ce6907b87376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:54:43 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:42974
I20260806 23:54:43.966024 32548 tablet_service.cc:1514] Processing DeleteTablet for tablet 20dd24e88ea44de8b2a4ce6907b87376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:54:43 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:56726
I20260806 23:54:43.966150 32741 tablet_service.cc:1514] Processing DeleteTablet for tablet 20dd24e88ea44de8b2a4ce6907b87376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:54:43 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:53500
I20260806 23:54:43.966656 425 tablet_replica.cc:330] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: stopping tablet replica
I20260806 23:54:43.966706 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 32226
I20260806 23:54:43.967011 425 raft_consensus.cc:2229] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:54:43.967423 425 raft_consensus.cc:2258] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:54:43.973002 425 ts_tablet_manager.cc:1905] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:54:43.976583 425 ts_tablet_manager.cc:1918] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:54:43.976706 425 log.cc:1196] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/20dd24e88ea44de8b2a4ce6907b87376
I20260806 23:54:43.977094 425 ts_tablet_manager.cc:1939] T 20dd24e88ea44de8b2a4ce6907b87376 P 20f9fba3083b46d3b6dd81813084bcef: Deleting consensus metadata
I20260806 23:54:43.978927 32146 catalog_manager.cc:4841] TS 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501): tablet 20dd24e88ea44de8b2a4ce6907b87376 (table test-table [id=ef20830db83e4b098fbb4b93d1a11cc4]) successfully deleted
I20260806 23:54:43.980131 423 tablet_replica.cc:330] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: stopping tablet replica
I20260806 23:54:43.980607 423 raft_consensus.cc:2229] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:54:43.981084 423 raft_consensus.cc:2258] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:54:43.982860 423 ts_tablet_manager.cc:1905] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:54:43.985736 423 ts_tablet_manager.cc:1918] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:54:43.985847 423 log.cc:1196] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/20dd24e88ea44de8b2a4ce6907b87376
I20260806 23:54:43.986146 423 ts_tablet_manager.cc:1939] T 20dd24e88ea44de8b2a4ce6907b87376 P 83052a98317a496ab42ba3b97f224ca9: Deleting consensus metadata
I20260806 23:54:43.987536 32146 catalog_manager.cc:4841] TS 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691): tablet 20dd24e88ea44de8b2a4ce6907b87376 (table test-table [id=ef20830db83e4b098fbb4b93d1a11cc4]) successfully deleted
W20260806 23:54:43.988483 32152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:43375 (error 108)
I20260806 23:54:43.989665 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 32459
I20260806 23:54:44.001593 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:44.001157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:43375 (local address 127.0.0.1:42984)
0806 23:54:44.001388 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.001484 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:43375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260806 23:54:44.002058 32152 catalog_manager.cc:4584] TS 99315324fc26442398b7cc25f29dafdc (127.10.225.129:43375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20dd24e88ea44de8b2a4ce6907b87376: Network error: Client connection negotiation failed: client connection to 127.10.225.129:43375: connect: Connection refused (error 111)
I20260806 23:54:44.009527 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 32651
I20260806 23:54:44.023499 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 32122
I20260806 23:54:44.036626 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34845
--webserver_interface=127.10.225.190
--webserver_port=34315
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:54:44.157123 428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:44.157567 428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:44.157720 428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:44.163594 428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:54:44.163815 428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:44.163866 428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:44.163887 428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:54:44.163915 428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:54:44.171365 428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=34315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:44.172627 428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:44.173672 428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:44.182344 439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:44.182344 437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:44.182348 436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:44.182633 428 server_base.cc:1034] running on GCE node
I20260806 23:54:44.183214 428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:44.184423 428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:44.185595 428 hybrid_clock.cc:648] HybridClock initialized: now 1786060484185573 us; error 38 us; skew 500 ppm
Aug 06 23:54:44 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:44.188488 428 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:54:44.190047 428 webserver.cc:460] Webserver started at http://127.10.225.190:34315/ using document root <none> and password file <none>
I20260806 23:54:44.190804 428 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:44.190898 428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:44.195480 428 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:54:44.198077 450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:44.200102 428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:44.200254 428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "1c0e22cc086c45d7a023d019fb695d8e"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "759821544b1c1e530156853b737236c5"
server_key_iv: "332c0a714b645d938c0338e8d617b426"
server_key_version: "encryptionkey@0"
I20260806 23:54:44.200856 428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:44.222395 428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:44.227152 428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:44.227391 428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:44.236881 428 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:34845
I20260806 23:54:44.237164 528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:34845 every 8 connection(s)
I20260806 23:54:44.238123 428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:54:44.242810 529 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:54:44.244583 529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Bootstrap starting.
I20260806 23:54:44.245883 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 428
I20260806 23:54:44.246387 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:43375
--local_ip_for_outbound_sockets=127.10.225.129
--tserver_master_addrs=127.10.225.190:34845
--webserver_port=39727
--webserver_interface=127.10.225.129
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:54:44.263334 529 log.cc:824] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:44.274961 529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:54:44.275364 529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e: Bootstrap complete.
I20260806 23:54:44.279764 529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:44.280196 529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0e22cc086c45d7a023d019fb695d8e, State: Initialized, Role: FOLLOWER
I20260806 23:54:44.280681 529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:44.280809 529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:54:44.280871 529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:54:44.280966 529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 2 FOLLOWER]: Advancing to term 3
I20260806 23:54:44.283092 529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:44.283445 529 leader_election.cc:302] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c0e22cc086c45d7a023d019fb695d8e; no voters:
I20260806 23:54:44.283806 529 leader_election.cc:288] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 23:54:44.283917 535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 23:54:44.284121 535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [term 3 LEADER]: Becoming Leader. State: Replica: 1c0e22cc086c45d7a023d019fb695d8e, State: Running, Role: LEADER
I20260806 23:54:44.284433 535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } }
I20260806 23:54:44.285116 529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:54:44.285529 535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0e22cc086c45d7a023d019fb695d8e. Latest consensus state: current_term: 3 leader_uuid: "1c0e22cc086c45d7a023d019fb695d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } } }
I20260806 23:54:44.285653 535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:44.286033 540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:54:44.286046 537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1c0e22cc086c45d7a023d019fb695d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0e22cc086c45d7a023d019fb695d8e" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 34845 } } }
I20260806 23:54:44.286283 537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0e22cc086c45d7a023d019fb695d8e [sys.catalog]: This master's current role is: LEADER
I20260806 23:54:44.291389 540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:54:44.292145 540 catalog_manager.cc:1259] Loaded cluster ID: 8874634a975d4d95bafe3b1aa40827fb
I20260806 23:54:44.292196 540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 23:54:44.293435 540 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd3150cf9ad google::LogMessage::Fail() at ??:0
@ 0x7fd3150d288a google::LogMessage::SendToLog() at ??:0
@ 0x7fd3150cf4de google::LogMessage::Flush() at ??:0
@ 0x7fd3150d0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd31fa12fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd31fa13d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd31fa10579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd31fa54640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd3173213d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd317321c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd3173233f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd317312512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd3189566db start_thread at ??:0
@ 0x7fd313bcd61f clone at ??:0
W20260806 23:54:44.384006 532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:44.384527 532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:44.384760 532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:44.389753 532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:44.389847 532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:44.389932 532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:54:44.396124 532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:43375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=39727
--tserver_master_addrs=127.10.225.190:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:44.397286 532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:44.398334 532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:44.408165 600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:44.408252 599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:44.408169 603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:44.408788 532 server_base.cc:1034] running on GCE node
I20260806 23:54:44.409193 532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:44.409864 532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:44.411052 532 hybrid_clock.cc:648] HybridClock initialized: now 1786060484411028 us; error 42 us; skew 500 ppm
Aug 06 23:54:44 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:44.414105 532 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:54:44.415215 532 webserver.cc:460] Webserver started at http://127.10.225.129:39727/ using document root <none> and password file <none>
I20260806 23:54:44.415830 532 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:44.415907 532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:44.420584 532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260806 23:54:44.423920 612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:44.425557 532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:54:44.425709 532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "99315324fc26442398b7cc25f29dafdc"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "47075a54a51e19ee713c4b3f2b093366"
server_key_iv: "6ef231a747360e4906a6b2bbbddf5d70"
server_key_version: "encryptionkey@0"
I20260806 23:54:44.426304 532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:44.437036 532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:44.440605 532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:44.440799 532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:44.441426 532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:44.442600 622 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 23:54:44.448611 532 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 23:54:44.448693 532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 23:54:44.448773 532 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 23:54:44.450608 532 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 23:54:44.450659 532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 23:54:44.451016 622 tablet_bootstrap.cc:492] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Bootstrap starting.
I20260806 23:54:44.461884 532 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:43375
I20260806 23:54:44.462201 774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:43375 every 8 connection(s)
I20260806 23:54:44.463289 532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:54:44.465978 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 532
I20260806 23:54:44.466485 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:45691
--local_ip_for_outbound_sockets=127.10.225.130
--tserver_master_addrs=127.10.225.190:34845
--webserver_port=39225
--webserver_interface=127.10.225.130
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:54:44.477217 622 log.cc:824] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Log is configured to *not* fsync() on all Append() calls
I20260806 23:54:44.487802 622 tablet_bootstrap.cc:492] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:54:44.488300 622 tablet_bootstrap.cc:492] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Bootstrap complete.
I20260806 23:54:44.489784 622 ts_tablet_manager.cc:1397] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.015s
I20260806 23:54:44.493322 622 raft_consensus.cc:348] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:44.493930 622 raft_consensus.cc:729] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99315324fc26442398b7cc25f29dafdc, State: Initialized, Role: FOLLOWER
I20260806 23:54:44.494495 622 consensus_queue.cc:260] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:44.495617 622 ts_tablet_manager.cc:1428] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260806 23:54:44.534330 778 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:54:44.467302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:54139)
0806 23:54:44.468202 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.468267 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:44.469726 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:44.533320 (+ 63594us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":282,"threads_started":1}
W20260806 23:54:44.534893 776 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:34845 (0 consecutive failures): Network error: Failed to ping master at 127.10.225.190:34845: Client connection negotiation failed: client connection to 127.10.225.190:34845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 23:54:44.535749 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:44.535425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:51773)
0806 23:54:44.535570 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.535647 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 23:54:44.536450 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:44.536224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:55441)
0806 23:54:44.536332 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.536383 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 23:54:44.536635 776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:54:44.596472 779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:44.596755 779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:44.596815 779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:44.601065 779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:44.601130 779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:44.601199 779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:54:44.606712 779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:45691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=39225
--tserver_master_addrs=127.10.225.190:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:44.607846 779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:44.608892 779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:44.618644 791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:44.618660 779 server_base.cc:1034] running on GCE node
W20260806 23:54:44.618654 788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:44.618654 789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:44.619305 779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:44.619993 779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:44.621171 779 hybrid_clock.cc:648] HybridClock initialized: now 1786060484621146 us; error 37 us; skew 500 ppm
Aug 06 23:54:44 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:44.624060 779 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:54:44.625039 779 webserver.cc:460] Webserver started at http://127.10.225.130:39225/ using document root <none> and password file <none>
I20260806 23:54:44.625612 779 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:44.625676 779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:44.629756 779 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:54:44.632131 799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:44.633877 779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:54:44.633997 779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "83052a98317a496ab42ba3b97f224ca9"
format_stamp: "Formatted at 2026-08-06 23:54:41 on dist-test-slave-nj21"
server_key: "a932a9ce2f9b850e0541519e3c0053f9"
server_key_iv: "2526c2894d066463fc56f01f25013256"
server_key_version: "encryptionkey@0"
I20260806 23:54:44.634603 779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:44.660322 779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:44.664743 779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:44.664963 779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:44.665881 779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:44.667125 779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:44.667198 779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:44.667270 779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:44.667305 779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:44.677057 779 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:45691
I20260806 23:54:44.677078 971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:45691 every 8 connection(s)
I20260806 23:54:44.678126 779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:54:44.680492 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 779
I20260806 23:54:44.681157 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:46501
--local_ip_for_outbound_sockets=127.10.225.131
--tserver_master_addrs=127.10.225.190:34845
--webserver_port=44545
--webserver_interface=127.10.225.131
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:54:44.683167 974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:44.680366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:56077)
0806 23:54:44.681072 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.681379 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":130,"threads_started":1}
W20260806 23:54:44.683717 972 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:34845 (0 consecutive failures): Network error: Failed to ping master at 127.10.225.190:34845: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
I20260806 23:54:44.684446 974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:44.684164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:41381)
0806 23:54:44.684323 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.684376 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 23:54:44.685061 974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:44.684878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:44795)
0806 23:54:44.684972 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.685011 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 23:54:44.685223 972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:54:44.802682 975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:54:44.802976 975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:54:44.803022 975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:54:44.808070 975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:54:44.808182 975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:54:44.808271 975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:54:44.815524 975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:46501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=44545
--tserver_master_addrs=127.10.225.190:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:54:44.816706 975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:54:44.817744 975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:54:44.828075 981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:44.828092 983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:54:44.828092 985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:54:44.828362 975 server_base.cc:1034] running on GCE node
I20260806 23:54:44.828975 975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:54:44.829766 975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:54:44.830937 975 hybrid_clock.cc:648] HybridClock initialized: now 1786060484830917 us; error 33 us; skew 500 ppm
Aug 06 23:54:44 dist-test-slave-nj21 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:54:44.834318 975 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:54:44.835543 975 webserver.cc:460] Webserver started at http://127.10.225.131:44545/ using document root <none> and password file <none>
I20260806 23:54:44.836299 975 fs_manager.cc:359] Metadata directory not provided
I20260806 23:54:44.836376 975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:54:44.841500 975 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:54:44.844012 997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:44.845378 975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:54:44.845496 975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "20f9fba3083b46d3b6dd81813084bcef"
format_stamp: "Formatted at 2026-08-06 23:54:42 on dist-test-slave-nj21"
server_key: "8cedc420f6fd7cd58e13047701630ade"
server_key_iv: "cbeea9282cadfbca7940e752a1fd8bf8"
server_key_version: "encryptionkey@0"
I20260806 23:54:44.846135 975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:54:44.870108 975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:54:44.873337 975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:54:44.873524 975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:54:44.874167 975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:54:44.875025 975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:54:44.875072 975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:44.875113 975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:54:44.875142 975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:54:44.885589 975 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:46501
I20260806 23:54:44.885632 1165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:46501 every 8 connection(s)
I20260806 23:54:44.886648 975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:54:44.890774 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 975
I20260806 23:54:44.890904 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:44.889006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:42817)
0806 23:54:44.889569 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.889703 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":126,"threads_started":1}
W20260806 23:54:44.891379 1166 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:34845 (0 consecutive failures): Network error: Failed to ping master at 127.10.225.190:34845: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
I20260806 23:54:44.891999 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:44.891776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:49039)
0806 23:54:44.891884 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.891940 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 23:54:44.892568 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:44.892364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:57563)
0806 23:54:44.892464 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:44.892512 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 23:54:44.892719 1166 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 23:54:45.538184 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:45.537560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:50693)
0806 23:54:45.537966 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:45.538044 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":105,"threads_started":1}
I20260806 23:54:45.686419 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:45.685920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:39231)
0806 23:54:45.686228 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:45.686298 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":76,"threads_started":1}
I20260806 23:54:45.894165 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:45.893505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:46471)
0806 23:54:45.893821 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:45.893881 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
I20260806 23:54:46.080498 1179 raft_consensus.cc:482] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:54:46.080732 1179 raft_consensus.cc:504] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:46.081874 1179 leader_election.cc:288] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501), 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691)
Aug 06 23:54:46 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.131@KRBTEST.COM
Aug 06 23:54:46 dist-test-slave-nj21 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060484, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.130@KRBTEST.COM
I20260806 23:54:46.095630 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:46.082200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:46501 (local address 127.10.225.129:44303)
0806 23:54:46.082489 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:46.082506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:46.082639 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:46.084588 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:46.084595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:46.085057 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:46.085495 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:46.085511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:46.086782 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:46.086785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:46.087821 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:46.087833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:46.088519 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:46.088523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:46.088764 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:46.089424 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:46.089436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:46.091114 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:46.093223 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:46.093229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:46.093243 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:46.093555 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:46.093897 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:46.093900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:46.093903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:46.094041 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:46.094682 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:46.094689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:46.094910 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 23:54:46.095084 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 23:54:46.095423 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":151,"spinlock_wait_cycles":1920,"thread_start_us":71,"threads_started":1}
I20260806 23:54:46.095656 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:46.082446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:44303 (local address 127.10.225.131:46501)
0806 23:54:46.082915 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:46.082944 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:46.082991 (+ 47us) server_negotiation.cc:407] Connection header received
0806 23:54:46.083753 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:46.083760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:46.084185 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:46.085121 (+ 936us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:46.085664 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:46.086638 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:46.087939 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:46.088397 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:46.088553 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:46.091353 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:46.091358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:46.091370 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:46.091408 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:46.093092 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:46.093675 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:46.093681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:46.093685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:46.093802 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:46.094137 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:46.094140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:46.094141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:46.094562 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:46.094686 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:46.095221 (+ 535us) server_negotiation.cc:299] Negotiation successful
0806 23:54:46.095489 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":134,"threads_started":1}
I20260806 23:54:46.096413 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:46.082563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:45691 (local address 127.10.225.129:35897)
0806 23:54:46.082856 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:46.082869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:54:46.082970 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:54:46.084633 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:54:46.084636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:54:46.085103 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:54:46.085497 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:46.085511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:46.086786 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:46.086789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:46.087826 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:54:46.087836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:54:46.088526 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:54:46.088530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:54:46.088765 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:46.089410 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:54:46.089427 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:54:46.091330 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:54:46.093294 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:46.093297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:46.093299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:46.093791 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:46.094397 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:54:46.094401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:54:46.094403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:54:46.094510 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:54:46.095590 (+ 1080us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:54:46.095594 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:54:46.095938 (+ 344us) client_negotiation.cc:769] Sending connection context
0806 23:54:46.096061 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 23:54:46.096190 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":141,"spinlock_wait_cycles":17408,"thread_start_us":67,"threads_started":1}
I20260806 23:54:46.097178 1185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:46.082787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:35897 (local address 127.10.225.130:45691)
0806 23:54:46.083129 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0806 23:54:46.083149 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:54:46.083178 (+ 29us) server_negotiation.cc:407] Connection header received
0806 23:54:46.083850 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:54:46.083857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:54:46.084186 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:54:46.084497 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:54:46.085665 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:46.086636 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:46.087937 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:54:46.088396 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:54:46.088553 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:54:46.091522 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:54:46.091527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:54:46.091538 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:54:46.091567 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:54:46.093182 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:46.094043 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:46.094052 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:46.094060 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:46.094268 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:54:46.094612 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:54:46.094615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:54:46.094616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:54:46.095323 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:54:46.095472 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 23:54:46.096723 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0806 23:54:46.097001 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":135,"threads_started":1}
W20260806 23:54:46.098460 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
W20260806 23:54:46.099567 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
I20260806 23:54:46.099642 614 leader_election.cc:302] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99315324fc26442398b7cc25f29dafdc; no voters: 20f9fba3083b46d3b6dd81813084bcef, 83052a98317a496ab42ba3b97f224ca9
I20260806 23:54:46.099907 1179 raft_consensus.cc:2726] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:54:46.539626 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:46.539245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:38479)
0806 23:54:46.539421 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:46.539504 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 23:54:46.687965 1186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:46.687439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:41191)
0806 23:54:46.687787 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:46.687852 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260806 23:54:46.895857 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:46.895283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:42677)
0806 23:54:46.895621 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:46.895740 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":109,"threads_started":1}
I20260806 23:54:47.541281 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:47.540695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:41395)
0806 23:54:47.541064 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:47.541145 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":84,"threads_started":1}
I20260806 23:54:47.604031 1189 raft_consensus.cc:482] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:54:47.604190 1189 raft_consensus.cc:504] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:47.604666 1189 leader_election.cc:288] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501), 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691)
W20260806 23:54:47.605567 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
W20260806 23:54:47.605718 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
I20260806 23:54:47.605770 614 leader_election.cc:302] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99315324fc26442398b7cc25f29dafdc; no voters: 20f9fba3083b46d3b6dd81813084bcef, 83052a98317a496ab42ba3b97f224ca9
I20260806 23:54:47.606060 1189 raft_consensus.cc:2726] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:54:47.691078 1192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:47.690551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:51035)
0806 23:54:47.690850 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:47.690941 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260806 23:54:47.897109 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:47.896702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:54171)
0806 23:54:47.896953 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:47.897010 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260806 23:54:48.542734 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:48.542224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:48093)
0806 23:54:48.542540 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:48.542611 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":91,"threads_started":1}
I20260806 23:54:48.692693 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:48.692023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:48889)
0806 23:54:48.692448 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:48.692550 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":84,"threads_started":1}
I20260806 23:54:48.898890 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:48.898274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:57537)
0806 23:54:48.898682 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:48.898759 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260806 23:54:49.341928 1200 raft_consensus.cc:482] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:54:49.342093 1200 raft_consensus.cc:504] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:49.342494 1200 leader_election.cc:288] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501), 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691)
W20260806 23:54:49.343329 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
W20260806 23:54:49.343475 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
I20260806 23:54:49.343523 614 leader_election.cc:302] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99315324fc26442398b7cc25f29dafdc; no voters: 20f9fba3083b46d3b6dd81813084bcef, 83052a98317a496ab42ba3b97f224ca9
I20260806 23:54:49.343711 1200 raft_consensus.cc:2726] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:54:49.544229 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:49.543737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:49273)
0806 23:54:49.544070 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:49.544129 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":76,"threads_started":1}
I20260806 23:54:49.694190 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:49.693706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:41839)
0806 23:54:49.694017 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:49.694084 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
I20260806 23:54:49.900696 1206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:49.900076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:43209)
0806 23:54:49.900484 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:49.900564 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":117,"threads_started":1}
I20260806 23:54:50.545665 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:50.545206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:45393)
0806 23:54:50.545495 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:50.545561 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260806 23:54:50.695600 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:50.695084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:40531)
0806 23:54:50.695417 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:50.695494 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":82,"threads_started":1}
I20260806 23:54:50.902488 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:50.901920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:48551)
0806 23:54:50.902309 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:50.902376 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260806 23:54:51.547230 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:51.546545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:60241)
0806 23:54:51.546994 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:51.547092 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":84,"threads_started":1}
I20260806 23:54:51.696961 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:51.696460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:36645)
0806 23:54:51.696790 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:51.696848 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":84,"threads_started":1}
I20260806 23:54:51.904150 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:51.903578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:49901)
0806 23:54:51.903980 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:51.904045 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":141,"threads_started":1}
I20260806 23:54:52.548775 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:52.548258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:33689)
0806 23:54:52.548596 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:52.548664 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260806 23:54:52.698568 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:52.697842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:33869)
0806 23:54:52.698228 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:52.698401 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":91,"threads_started":1}
I20260806 23:54:52.905525 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:52.905070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:39495)
0806 23:54:52.905353 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:52.905419 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260806 23:54:53.462327 1219 raft_consensus.cc:482] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:54:53.462637 1219 raft_consensus.cc:504] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:53.463187 1219 leader_election.cc:288] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501), 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691)
W20260806 23:54:53.464344 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
W20260806 23:54:53.464488 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
I20260806 23:54:53.464542 614 leader_election.cc:302] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99315324fc26442398b7cc25f29dafdc; no voters: 20f9fba3083b46d3b6dd81813084bcef, 83052a98317a496ab42ba3b97f224ca9
I20260806 23:54:53.464704 1219 raft_consensus.cc:2726] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:54:53.550251 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:53.549677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:50937)
0806 23:54:53.550037 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:53.550127 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":87,"threads_started":1}
I20260806 23:54:53.699975 1222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:53.699499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:48049)
0806 23:54:53.699804 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:53.699865 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260806 23:54:53.906951 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:53.906421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:33015)
0806 23:54:53.906747 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:53.906826 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
I20260806 23:54:54.551744 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:54.551284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:34571)
0806 23:54:54.551540 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:54.551608 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260806 23:54:54.701738 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:54.701084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:51947)
0806 23:54:54.701496 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:54.701600 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":89,"threads_started":1}
I20260806 23:54:54.908363 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:54.907893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:37305)
0806 23:54:54.908185 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:54.908250 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
I20260806 23:54:55.553138 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:55.552636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:55769)
0806 23:54:55.552970 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:55.553035 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
I20260806 23:54:55.703532 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:55.702933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:55153)
0806 23:54:55.703300 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:55.703388 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260806 23:54:55.909667 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:55.909197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:53115)
0806 23:54:55.909500 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:55.909564 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
I20260806 23:54:56.452744 1235 raft_consensus.cc:482] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:54:56.452886 1235 raft_consensus.cc:504] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f9fba3083b46d3b6dd81813084bcef" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46501 } } peers { permanent_uuid: "83052a98317a496ab42ba3b97f224ca9" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 45691 } } peers { permanent_uuid: "99315324fc26442398b7cc25f29dafdc" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 43375 } }
I20260806 23:54:56.453332 1235 leader_election.cc:288] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501), 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691)
W20260806 23:54:56.454233 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83052a98317a496ab42ba3b97f224ca9 (127.10.225.130:45691): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
W20260806 23:54:56.454440 614 leader_election.cc:341] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20f9fba3083b46d3b6dd81813084bcef (127.10.225.131:46501): Not found: Tablet not found: 20dd24e88ea44de8b2a4ce6907b87376
I20260806 23:54:56.454491 614 leader_election.cc:302] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99315324fc26442398b7cc25f29dafdc; no voters: 20f9fba3083b46d3b6dd81813084bcef, 83052a98317a496ab42ba3b97f224ca9
I20260806 23:54:56.454687 1235 raft_consensus.cc:2726] T 20dd24e88ea44de8b2a4ce6907b87376 P 99315324fc26442398b7cc25f29dafdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:54:56.554665 1237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:56.554084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:49971)
0806 23:54:56.554454 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:56.554551 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260806 23:54:56.705080 1238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:56.704595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:35659)
0806 23:54:56.704904 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:56.704971 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260806 23:54:56.910965 1239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:56.910498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:37783)
0806 23:54:56.910780 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:56.910839 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260806 23:54:57.556025 1240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:57.555535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:59805)
0806 23:54:57.555855 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:57.555917 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":122,"threads_started":1}
I20260806 23:54:57.706482 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:57.706027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:49055)
0806 23:54:57.706307 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:57.706368 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260806 23:54:57.912388 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:57.911940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:56705)
0806 23:54:57.912223 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:57.912278 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20260806 23:54:58.557544 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:58.556935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:46673)
0806 23:54:58.557358 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:58.557435 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":103,"threads_started":1}
I20260806 23:54:58.708253 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:58.707520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:45703)
0806 23:54:58.707997 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:58.708102 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":93,"threads_started":1}
I20260806 23:54:58.913801 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:58.913309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:53637)
0806 23:54:58.913614 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:58.913686 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260806 23:54:59.559228 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:59.558788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.129:57253)
0806 23:54:59.559066 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:59.559129 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260806 23:54:59.709816 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:59.709388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.130:58639)
0806 23:54:59.709647 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:59.709708 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260806 23:54:59.893953 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 532
I20260806 23:54:59.909730 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 779
I20260806 23:54:59.924623 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:54:59.922850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:34845 (local address 127.10.225.131:46821)
0806 23:54:59.924386 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0806 23:54:59.924477 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1426,"thread_start_us":428,"threads_started":1}
I20260806 23:54:59.931802 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 975
2026-08-06T23:54:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20084 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:55:00 dist-test-slave-nj21 krb5kdc[1255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:55:00 dist-test-slave-nj21 krb5kdc[1255](info): set up 2 sockets
Aug 06 23:55:00 dist-test-slave-nj21 krb5kdc[1255](info): commencing operation
krb5kdc: starting...
W20260806 23:55:01.110234 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:55:01 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:55:01.251308 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41925
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:41925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:55:01.371873 1274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:01.372167 1274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:01.372210 1274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:01.376444 1274 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 23:55:01.376509 1274 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 23:55:01.376524 1274 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 23:55:01.376538 1274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:55:01.376554 1274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:01.376571 1274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:01.376585 1274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:55:01.376600 1274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:55:01.382066 1274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:41925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:01.383155 1274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:01.384223 1274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:55:01.394196 1274 server_base.cc:1034] running on GCE node
W20260806 23:55:01.394142 1281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:01.394181 1285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:01.394142 1283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:01.395069 1274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:01.396257 1274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:01.397440 1274 hybrid_clock.cc:648] HybridClock initialized: now 1786060501397426 us; error 30 us; skew 500 ppm
Aug 06 23:55:01 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060501, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:01.400424 1274 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:55:01.401396 1274 webserver.cc:460] Webserver started at http://127.10.225.190:40659/ using document root <none> and password file <none>
I20260806 23:55:01.401999 1274 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:01.402062 1274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:01.402268 1274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:55:01.404465 1274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "c736983ecca04e6780a5cad3743ec0a1"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "1ba90988054f97e29eaf788cded8e199"
server_key_iv: "f5292094dcaf5934024b2e8f9ff88c5e"
server_key_version: "encryptionkey@0"
I20260806 23:55:01.405048 1274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "c736983ecca04e6780a5cad3743ec0a1"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "1ba90988054f97e29eaf788cded8e199"
server_key_iv: "f5292094dcaf5934024b2e8f9ff88c5e"
server_key_version: "encryptionkey@0"
I20260806 23:55:01.409128 1274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:55:01.411662 1297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:01.413380 1274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:55:01.413522 1274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "c736983ecca04e6780a5cad3743ec0a1"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "1ba90988054f97e29eaf788cded8e199"
server_key_iv: "f5292094dcaf5934024b2e8f9ff88c5e"
server_key_version: "encryptionkey@0"
I20260806 23:55:01.413642 1274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:01.429570 1274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:01.432771 1274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:01.432972 1274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:01.440171 1274 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:41925
I20260806 23:55:01.440523 1371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:41925 every 8 connection(s)
I20260806 23:55:01.441345 1274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:55:01.444345 1374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:55:01.449667 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 1274
I20260806 23:55:01.449841 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:55:01.450225 11142 external_mini_cluster.cc:1428] Setting key 318323a22f65bdc8b48552a6f4f2cbb3
I20260806 23:55:01.450325 1374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Bootstrap starting.
I20260806 23:55:01.453294 1374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Neither blocks nor log segments found. Creating new log.
I20260806 23:55:01.454274 1374 log.cc:824] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:01.456452 1374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: No bootstrap required, opened a new log
Aug 06 23:55:01 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:55:01.460875 1374 raft_consensus.cc:348] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:01.461118 1374 raft_consensus.cc:374] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:55:01.461189 1374 raft_consensus.cc:729] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c736983ecca04e6780a5cad3743ec0a1, State: Initialized, Role: FOLLOWER
I20260806 23:55:01.461654 1374 consensus_queue.cc:260] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:01.461796 1374 raft_consensus.cc:388] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:55:01.461863 1374 raft_consensus.cc:482] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:55:01.461974 1374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:55:01.462895 1374 raft_consensus.cc:504] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:01.463294 1374 leader_election.cc:302] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c736983ecca04e6780a5cad3743ec0a1; no voters:
I20260806 23:55:01.463649 1374 leader_election.cc:288] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:55:01.463796 1380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:55:01.464030 1380 raft_consensus.cc:686] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 1 LEADER]: Becoming Leader. State: Replica: c736983ecca04e6780a5cad3743ec0a1, State: Running, Role: LEADER
I20260806 23:55:01.464416 1380 consensus_queue.cc:237] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:01.465092 1374 sys_catalog.cc:564] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:55:01.466647 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:01.451954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.10.225.190:41925)
0806 23:55:01.452443 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:01.452463 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:01.452506 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:55:01.453381 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:01.453406 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:01.453847 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:01.454211 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:01.454962 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.456072 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.457232 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.457728 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.457877 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:01.460669 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:01.460677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:01.460693 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:01.460722 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:01.462496 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:01.462907 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:01.462914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:01.462920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:01.463041 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:01.463527 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:01.463542 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:01.463544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:01.464237 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:01.464467 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:01.465135 (+ 668us) server_negotiation.cc:299] Negotiation successful
0806 23:55:01.465476 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":132,"threads_started":1}
I20260806 23:55:01.468169 1381 sys_catalog.cc:455] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c736983ecca04e6780a5cad3743ec0a1. Latest consensus state: current_term: 1 leader_uuid: "c736983ecca04e6780a5cad3743ec0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } } }
I20260806 23:55:01.468406 1381 sys_catalog.cc:458] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:55:01.468370 1382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c736983ecca04e6780a5cad3743ec0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } } }
I20260806 23:55:01.468832 1382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: This master's current role is: LEADER
W20260806 23:55:01.470991 1399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:55:01.471125 1399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:55:01.471225 1400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:55:01.472357 1400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:55:01.479470 1400 catalog_manager.cc:1347] Generated new cluster ID: a7259a69b5b348d28ad419d91ecdb1e5
I20260806 23:55:01.479578 1400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:55:01.486861 1400 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:55:01.488037 1400 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:55:01.494127 1400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Generated new TSK 0
I20260806 23:55:01.494855 1400 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:55:01.531904 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41925
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:55:01.650902 1408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:01.651336 1408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:01.651412 1408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:01.656476 1408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:01.656576 1408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:01.656677 1408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:55:01.662561 1408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:01.663762 1408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:01.664826 1408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:01.675114 1420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:01.675114 1422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:01.675117 1419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:01.675748 1408 server_base.cc:1034] running on GCE node
I20260806 23:55:01.676517 1408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:01.677342 1408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:01.678469 1408 hybrid_clock.cc:648] HybridClock initialized: now 1786060501678462 us; error 26 us; skew 500 ppm
Aug 06 23:55:01 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060501, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:01.681787 1408 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:55:01.682900 1408 webserver.cc:460] Webserver started at http://127.10.225.129:38163/ using document root <none> and password file <none>
I20260806 23:55:01.683691 1408 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:01.683787 1408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:01.684042 1408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:55:01.686378 1408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "80c3d4caae9b4a3ea0a0f126770b9a34"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "b7d8d57b098b0162662b3e3f70c56990"
server_key_iv: "735f47d3e71e29ab7555c8abb2a99060"
server_key_version: "encryptionkey@0"
I20260806 23:55:01.686962 1408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "80c3d4caae9b4a3ea0a0f126770b9a34"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "b7d8d57b098b0162662b3e3f70c56990"
server_key_iv: "735f47d3e71e29ab7555c8abb2a99060"
server_key_version: "encryptionkey@0"
I20260806 23:55:01.691031 1408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:55:01.693668 1438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:01.694958 1408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:55:01.695068 1408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "80c3d4caae9b4a3ea0a0f126770b9a34"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "b7d8d57b098b0162662b3e3f70c56990"
server_key_iv: "735f47d3e71e29ab7555c8abb2a99060"
server_key_version: "encryptionkey@0"
I20260806 23:55:01.695180 1408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:01.714375 1408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:01.717792 1408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:01.717995 1408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:01.718649 1408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:55:01.719535 1408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:55:01.719584 1408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:01.719626 1408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:55:01.719655 1408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:01.728936 1408 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:39259
I20260806 23:55:01.728961 1596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:39259 every 8 connection(s)
I20260806 23:55:01.729873 1408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:55:01.730401 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 1408
I20260806 23:55:01.730561 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:55:01.730862 11142 external_mini_cluster.cc:1428] Setting key 9df2ff5123a12b484c0114155aef43ba
Aug 06 23:55:01 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060501, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:55:01.746201 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:01.732379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:50423 (local address 127.10.225.190:41925)
0806 23:55:01.732530 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:01.732534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:01.733592 (+ 1058us) server_negotiation.cc:407] Connection header received
0806 23:55:01.735372 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:01.735376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:01.735430 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:01.735520 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:01.736791 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.737622 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.739109 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.739344 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.739429 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:01.742486 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:01.742490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:01.742492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:01.742517 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:01.743917 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:01.744459 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:01.744462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:01.744464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:01.744578 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:01.744931 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:01.744934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:01.744936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:01.745194 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:01.745496 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:01.745936 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 23:55:01.746067 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:55:01.747282 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:01.732850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:50423)
0806 23:55:01.733409 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:01.733467 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:01.735100 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:01.735752 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:01.735759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:01.736133 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:55:01.736640 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:01.736655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.737774 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.737778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:01.738978 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:01.738993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.739506 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.739511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:01.739762 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:01.740419 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:55:01.740442 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:55:01.742279 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:55:01.744042 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:01.744048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:01.744062 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:01.744358 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:01.744674 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:01.744677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:01.744679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:01.744819 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:01.745384 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:55:01.745390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:55:01.745653 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 23:55:01.745856 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 23:55:01.746214 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":128,"threads_started":1}
I20260806 23:55:01.748638 1598 heartbeater.cc:347] Connected to a master server at 127.10.225.190:41925
I20260806 23:55:01.748894 1598 heartbeater.cc:464] Registering TS with master...
I20260806 23:55:01.749408 1598 heartbeater.cc:511] Master 127.10.225.190:41925 requested a full tablet report, sending...
I20260806 23:55:01.751138 1320 ts_manager.cc:194] Registered new tserver with Master: 80c3d4caae9b4a3ea0a0f126770b9a34 (127.10.225.129:39259)
I20260806 23:55:01.752832 1320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:50423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:55:01.764156 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41925
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:55:01.892629 1607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:01.892899 1607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:01.892944 1607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:01.896919 1607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:01.896978 1607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:01.897038 1607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:55:01.902709 1607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:01.903780 1607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:01.904769 1607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:01.914639 1613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:01.914633 1615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:01.914639 1612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:01.915017 1607 server_base.cc:1034] running on GCE node
I20260806 23:55:01.915421 1607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:01.916112 1607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:01.917297 1607 hybrid_clock.cc:648] HybridClock initialized: now 1786060501917216 us; error 87 us; skew 500 ppm
Aug 06 23:55:01 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060501, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:01.920426 1607 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:55:01.921502 1607 webserver.cc:460] Webserver started at http://127.10.225.130:39135/ using document root <none> and password file <none>
I20260806 23:55:01.922103 1607 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:01.922293 1607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:01.922574 1607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:55:01.924801 1607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "8932d9a2a3d548a88298cbd1cd9f2476"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "208f4efd8498880f90ff784dc7377fa7"
server_key_iv: "d7c732eef9c3b98095d8b0700c822bca"
server_key_version: "encryptionkey@0"
I20260806 23:55:01.925360 1607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "8932d9a2a3d548a88298cbd1cd9f2476"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "208f4efd8498880f90ff784dc7377fa7"
server_key_iv: "d7c732eef9c3b98095d8b0700c822bca"
server_key_version: "encryptionkey@0"
I20260806 23:55:01.929648 1607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:55:01.932395 1628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:01.933890 1607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:55:01.934031 1607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "8932d9a2a3d548a88298cbd1cd9f2476"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "208f4efd8498880f90ff784dc7377fa7"
server_key_iv: "d7c732eef9c3b98095d8b0700c822bca"
server_key_version: "encryptionkey@0"
I20260806 23:55:01.934155 1607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:01.947512 1607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:01.950555 1607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:01.950740 1607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:01.951365 1607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:55:01.952268 1607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:55:01.952317 1607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:01.952376 1607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:55:01.952402 1607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:01.961342 1607 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:38811
I20260806 23:55:01.961380 1785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:38811 every 8 connection(s)
I20260806 23:55:01.962270 1607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:55:01.962463 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 1607
I20260806 23:55:01.962574 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:55:01.962890 11142 external_mini_cluster.cc:1428] Setting key 0aa564d7aeb2a225bad55267ed1d558d
Aug 06 23:55:01 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060501, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
I20260806 23:55:01.978080 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:01.964140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:51337 (local address 127.10.225.190:41925)
0806 23:55:01.964267 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:01.964272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:01.965301 (+ 1029us) server_negotiation.cc:407] Connection header received
0806 23:55:01.967147 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:01.967151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:01.967208 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:01.967281 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:01.968665 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.969263 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.970502 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.970727 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.970816 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:01.973710 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:01.973714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:01.973717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:01.973758 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:01.975856 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:01.976429 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:01.976432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:01.976434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:01.976524 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:01.976852 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:01.976856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:01.976857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:01.977073 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:01.977175 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:01.977750 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 23:55:01.977896 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:55:01.979355 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:01.964594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:51337)
0806 23:55:01.965115 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:01.965213 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:01.966895 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:01.967471 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:01.967478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:01.968003 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:55:01.968519 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:01.968533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.969392 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.969395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:01.970381 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:01.970392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:01.970876 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:01.970880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:01.971055 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:01.971702 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:55:01.971725 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:55:01.973366 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:55:01.976010 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:01.976017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:01.976034 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:01.976320 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:01.976612 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:01.976615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:01.976616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:01.976758 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:01.977198 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:55:01.977205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:55:01.977488 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 23:55:01.977673 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 23:55:01.978249 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":126,"threads_started":1}
I20260806 23:55:01.980667 1786 heartbeater.cc:347] Connected to a master server at 127.10.225.190:41925
I20260806 23:55:01.980917 1786 heartbeater.cc:464] Registering TS with master...
I20260806 23:55:01.981474 1786 heartbeater.cc:511] Master 127.10.225.190:41925 requested a full tablet report, sending...
I20260806 23:55:01.982596 1320 ts_manager.cc:194] Registered new tserver with Master: 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811)
I20260806 23:55:01.983207 1320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:51337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:55:01.994215 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41925
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:55:02.122879 1798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:02.123152 1798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:02.123198 1798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:02.127286 1798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:02.127351 1798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:02.127410 1798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:55:02.134663 1798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:02.135833 1798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:02.136879 1798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:02.147284 1808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:02.147360 1813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:02.147432 1798 server_base.cc:1034] running on GCE node
W20260806 23:55:02.147360 1810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:02.147953 1798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:02.148607 1798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:02.149775 1798 hybrid_clock.cc:648] HybridClock initialized: now 1786060502149755 us; error 33 us; skew 500 ppm
Aug 06 23:55:02 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060502, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:02.152635 1798 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:55:02.153553 1798 webserver.cc:460] Webserver started at http://127.10.225.131:39625/ using document root <none> and password file <none>
I20260806 23:55:02.154111 1798 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:02.154173 1798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:02.154377 1798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:55:02.156515 1798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "322cfb9e5c5146cf8d296d858c4b6a9a"
format_stamp: "Formatted at 2026-08-06 23:55:02 on dist-test-slave-nj21"
server_key: "eddf4fc75d1798373fef18e4824d1c04"
server_key_iv: "2d7735e7142741d788b8d55ea93e5777"
server_key_version: "encryptionkey@0"
I20260806 23:55:02.157058 1798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "322cfb9e5c5146cf8d296d858c4b6a9a"
format_stamp: "Formatted at 2026-08-06 23:55:02 on dist-test-slave-nj21"
server_key: "eddf4fc75d1798373fef18e4824d1c04"
server_key_iv: "2d7735e7142741d788b8d55ea93e5777"
server_key_version: "encryptionkey@0"
I20260806 23:55:02.161017 1798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:55:02.164095 1824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:02.165563 1798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 23:55:02.165707 1798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "322cfb9e5c5146cf8d296d858c4b6a9a"
format_stamp: "Formatted at 2026-08-06 23:55:02 on dist-test-slave-nj21"
server_key: "eddf4fc75d1798373fef18e4824d1c04"
server_key_iv: "2d7735e7142741d788b8d55ea93e5777"
server_key_version: "encryptionkey@0"
I20260806 23:55:02.165848 1798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:02.184563 1798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:02.188619 1798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:02.188817 1798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:02.189476 1798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:55:02.190349 1798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:55:02.190402 1798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:02.190443 1798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:55:02.190472 1798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:02.199860 1798 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:33835
I20260806 23:55:02.199904 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:33835 every 8 connection(s)
I20260806 23:55:02.200785 1798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:55:02.203050 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 1798
I20260806 23:55:02.203173 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:55:02.203524 11142 external_mini_cluster.cc:1428] Setting key c7f565ed773db21d15c532cea867362e
Aug 06 23:55:02 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060502, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:55:02.216048 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.202699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:48151 (local address 127.10.225.190:41925)
0806 23:55:02.202882 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:02.202886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:02.204328 (+ 1442us) server_negotiation.cc:407] Connection header received
0806 23:55:02.205945 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:02.205948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:02.206005 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:02.206076 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:02.207281 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.207943 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.209424 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.209660 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.209750 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.212409 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:02.212413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:02.212415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:02.212499 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:02.213936 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:02.214454 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:02.214458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:02.214460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:02.214538 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:02.214864 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:02.214867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:02.214868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:02.215065 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:02.215152 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:02.215714 (+ 562us) server_negotiation.cc:299] Negotiation successful
0806 23:55:02.215852 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 23:55:02.216991 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.203164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:48151)
0806 23:55:02.204146 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:02.204212 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:02.205735 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:02.206252 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:02.206260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:02.206636 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:55:02.207146 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.207160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.208063 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.208066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.209287 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.209302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.209822 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.209827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.209999 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.210627 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:55:02.210645 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:55:02.212233 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:55:02.214053 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:02.214059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:02.214077 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:02.214354 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:02.214626 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:02.214629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:02.214632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:02.214773 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:02.215169 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:55:02.215177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:55:02.215441 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 23:55:02.215622 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 23:55:02.216006 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":157,"threads_started":1}
I20260806 23:55:02.218324 1989 heartbeater.cc:347] Connected to a master server at 127.10.225.190:41925
I20260806 23:55:02.218562 1989 heartbeater.cc:464] Registering TS with master...
I20260806 23:55:02.219071 1989 heartbeater.cc:511] Master 127.10.225.190:41925 requested a full tablet report, sending...
I20260806 23:55:02.220090 1320 ts_manager.cc:194] Registered new tserver with Master: 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835)
I20260806 23:55:02.220688 1320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:48151
I20260806 23:55:02.230544 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:55:02.239969 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.232664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.10.225.190:41925)
0806 23:55:02.232786 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:02.232791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:02.232948 (+ 157us) server_negotiation.cc:407] Connection header received
0806 23:55:02.233084 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:02.233087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:02.233129 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:02.233194 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:02.233911 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.234428 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.235234 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.235450 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.235535 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.236912 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:02.236916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:02.236918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:02.236942 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:02.238308 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:02.238690 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:02.238693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:02.238694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:02.238804 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:02.239060 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:02.239063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:02.239065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:02.239297 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:02.239418 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:02.239652 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 23:55:02.239807 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 23:55:02.242940 1320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:55:02.246064 1320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:55:02.260991 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.255053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:38811 (local address 127.0.0.1:54618)
0806 23:55:02.255692 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:02.255706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:02.255796 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:02.256573 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:02.256579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:02.256604 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:55:02.256785 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.256793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.258474 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.258477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.259399 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.259410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.260472 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.260476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.260591 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.260628 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 23:55:02.260701 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 23:55:02.260809 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":5,"thread_start_us":117,"threads_started":1}
I20260806 23:55:02.261319 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.254956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:39259 (local address 127.0.0.1:59530)
0806 23:55:02.255538 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:02.255592 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:02.255742 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:02.256606 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:02.256609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:02.256621 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:55:02.256930 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.256937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.258430 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.258433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.259764 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.259777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.260849 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.260852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.260980 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.261037 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 23:55:02.261099 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 23:55:02.261174 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":291,"thread_start_us":107,"threads_started":1}
I20260806 23:55:02.261523 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.256265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.10.225.131:33835)
0806 23:55:02.256590 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:02.256605 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:02.256627 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:55:02.256701 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:02.256705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:02.256831 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:02.256961 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:55:02.257203 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.258102 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.259911 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.260710 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.260841 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.260981 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:02.261268 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 23:55:02.261381 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":132,"threads_started":1}
I20260806 23:55:02.261622 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.255767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:33835 (local address 127.0.0.1:59680)
0806 23:55:02.256025 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:02.256039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:02.256125 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:02.256962 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:02.256965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:02.256982 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:55:02.257076 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.257083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.258233 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.258237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.259782 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.259789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.260831 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.260834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.260919 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.261018 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 23:55:02.261086 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 23:55:02.261160 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":126,"thread_start_us":123,"threads_started":1}
I20260806 23:55:02.261791 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.255262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.10.225.130:38811)
0806 23:55:02.256049 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:02.256066 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:02.256091 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:55:02.256169 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:02.256174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:02.256393 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:02.256931 (+ 538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:55:02.257082 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.258303 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.259532 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.260347 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.261384 (+ 1037us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.261501 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:02.261551 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 23:55:02.261640 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":151,"threads_started":1}
I20260806 23:55:02.262363 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.255842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.10.225.129:39259)
0806 23:55:02.256235 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:02.256241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:02.256259 (+ 18us) server_negotiation.cc:407] Connection header received
0806 23:55:02.256323 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:02.256327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:02.256479 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:02.256708 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:55:02.257188 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.258303 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.259911 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.260710 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.261957 (+ 1247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.262083 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:02.262128 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 23:55:02.262214 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":121,"threads_started":1}
I20260806 23:55:02.264487 1698 tablet_service.cc:1467] Processing CreateTablet for tablet 01b3d8dda71740cbbe5822686548afb1 (DEFAULT_TABLE table=test-table [id=4db65be700294381815f67acea0dd6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:55:02.264595 1889 tablet_service.cc:1467] Processing CreateTablet for tablet 01b3d8dda71740cbbe5822686548afb1 (DEFAULT_TABLE table=test-table [id=4db65be700294381815f67acea0dd6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:55:02.264671 1500 tablet_service.cc:1467] Processing CreateTablet for tablet 01b3d8dda71740cbbe5822686548afb1 (DEFAULT_TABLE table=test-table [id=4db65be700294381815f67acea0dd6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:55:02.265511 1698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b3d8dda71740cbbe5822686548afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:55:02.265618 1500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b3d8dda71740cbbe5822686548afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:55:02.265628 1889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b3d8dda71740cbbe5822686548afb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:55:02.275405 2023 tablet_bootstrap.cc:492] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: Bootstrap starting.
I20260806 23:55:02.276136 2026 tablet_bootstrap.cc:492] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Bootstrap starting.
I20260806 23:55:02.278185 2026 tablet_bootstrap.cc:654] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Neither blocks nor log segments found. Creating new log.
I20260806 23:55:02.278515 2023 tablet_bootstrap.cc:654] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: Neither blocks nor log segments found. Creating new log.
I20260806 23:55:02.278985 2026 log.cc:824] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:02.279155 2023 log.cc:824] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:02.279740 2024 tablet_bootstrap.cc:492] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: Bootstrap starting.
I20260806 23:55:02.280932 2026 tablet_bootstrap.cc:492] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: No bootstrap required, opened a new log
I20260806 23:55:02.281092 2026 ts_tablet_manager.cc:1397] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 23:55:02.281509 2024 tablet_bootstrap.cc:654] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: Neither blocks nor log segments found. Creating new log.
I20260806 23:55:02.282550 2024 log.cc:824] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:02.284178 2023 tablet_bootstrap.cc:492] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: No bootstrap required, opened a new log
I20260806 23:55:02.284441 2024 tablet_bootstrap.cc:492] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: No bootstrap required, opened a new log
I20260806 23:55:02.284631 2024 ts_tablet_manager.cc:1397] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:55:02.284623 2023 ts_tablet_manager.cc:1397] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260806 23:55:02.285538 2026 raft_consensus.cc:348] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:02.285817 2026 raft_consensus.cc:374] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:55:02.285872 2026 raft_consensus.cc:729] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c3d4caae9b4a3ea0a0f126770b9a34, State: Initialized, Role: FOLLOWER
I20260806 23:55:02.286345 2026 consensus_queue.cc:260] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:02.287071 1598 heartbeater.cc:503] Master 127.10.225.190:41925 was elected leader, sending a full tablet report...
I20260806 23:55:02.288239 2026 ts_tablet_manager.cc:1428] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260806 23:55:02.288219 2024 raft_consensus.cc:348] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:02.288515 2024 raft_consensus.cc:374] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:55:02.288340 2023 raft_consensus.cc:348] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:02.288594 2023 raft_consensus.cc:374] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:55:02.288656 2023 raft_consensus.cc:729] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8932d9a2a3d548a88298cbd1cd9f2476, State: Initialized, Role: FOLLOWER
I20260806 23:55:02.288697 2024 raft_consensus.cc:729] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322cfb9e5c5146cf8d296d858c4b6a9a, State: Initialized, Role: FOLLOWER
I20260806 23:55:02.289095 2023 consensus_queue.cc:260] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:02.289148 2024 consensus_queue.cc:260] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:02.289811 1989 heartbeater.cc:503] Master 127.10.225.190:41925 was elected leader, sending a full tablet report...
I20260806 23:55:02.290083 1786 heartbeater.cc:503] Master 127.10.225.190:41925 was elected leader, sending a full tablet report...
I20260806 23:55:02.291286 2023 ts_tablet_manager.cc:1428] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 23:55:02.292369 2024 ts_tablet_manager.cc:1428] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 23:55:02.384905 2036 raft_consensus.cc:482] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:55:02.385228 2036 raft_consensus.cc:504] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:02.386806 2036 leader_election.cc:288] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80c3d4caae9b4a3ea0a0f126770b9a34 (127.10.225.129:39259), 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811)
I20260806 23:55:02.390547 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.387230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:47567 (local address 127.10.225.130:38811)
0806 23:55:02.387372 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:02.387376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:02.387391 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:55:02.387510 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:02.387513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:02.387559 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:02.387622 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:55:02.387922 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.388414 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.389429 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.389847 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.389958 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.390014 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:02.390221 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 23:55:02.390421 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 23:55:02.390529 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.387092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:38811 (local address 127.10.225.131:47567)
0806 23:55:02.387255 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:02.387289 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:02.387396 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:02.387648 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:02.387651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:02.387694 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:55:02.387810 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.387816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.388522 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.388526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.389305 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.389314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.389962 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.389965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.390107 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.390126 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 23:55:02.390192 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 23:55:02.390268 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":27}
I20260806 23:55:02.391467 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.387092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:39259 (local address 127.10.225.131:56085)
0806 23:55:02.387892 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:02.387907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:02.388063 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:02.388436 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:02.388440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:02.388462 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:55:02.388576 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.388581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.389273 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.389277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.390106 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:02.390115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.390940 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.390945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:02.391039 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.391064 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 23:55:02.391142 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 23:55:02.391203 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":329,"thread_start_us":225,"threads_started":1}
I20260806 23:55:02.391542 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.387230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:56085 (local address 127.10.225.129:39259)
0806 23:55:02.387497 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:02.387503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:02.388018 (+ 515us) server_negotiation.cc:407] Connection header received
0806 23:55:02.388261 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:02.388265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:02.388334 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:02.388408 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:55:02.388676 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.389170 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.390217 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.390757 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.390888 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.390934 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:02.391213 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 23:55:02.391409 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260806 23:55:02.391578 1723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3d8dda71740cbbe5822686548afb1" candidate_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" is_pre_election: true
I20260806 23:55:02.391997 1723 raft_consensus.cc:2454] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 322cfb9e5c5146cf8d296d858c4b6a9a in term 0.
I20260806 23:55:02.392328 1530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3d8dda71740cbbe5822686548afb1" candidate_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" is_pre_election: true
I20260806 23:55:02.392565 1829 leader_election.cc:302] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 322cfb9e5c5146cf8d296d858c4b6a9a, 8932d9a2a3d548a88298cbd1cd9f2476; no voters:
I20260806 23:55:02.392711 1530 raft_consensus.cc:2454] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 322cfb9e5c5146cf8d296d858c4b6a9a in term 0.
I20260806 23:55:02.392998 2036 raft_consensus.cc:2781] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:55:02.393115 2036 raft_consensus.cc:482] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:55:02.393175 2036 raft_consensus.cc:3037] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:55:02.394488 2036 raft_consensus.cc:504] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:02.394884 2036 leader_election.cc:288] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [CANDIDATE]: Term 1 election: Requested vote from peers 80c3d4caae9b4a3ea0a0f126770b9a34 (127.10.225.129:39259), 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811)
I20260806 23:55:02.395390 1723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3d8dda71740cbbe5822686548afb1" candidate_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8932d9a2a3d548a88298cbd1cd9f2476"
I20260806 23:55:02.395392 1530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3d8dda71740cbbe5822686548afb1" candidate_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34"
I20260806 23:55:02.395569 1723 raft_consensus.cc:3037] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:55:02.395570 1530 raft_consensus.cc:3037] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:55:02.397060 1530 raft_consensus.cc:2454] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 322cfb9e5c5146cf8d296d858c4b6a9a in term 1.
I20260806 23:55:02.397060 1723 raft_consensus.cc:2454] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 322cfb9e5c5146cf8d296d858c4b6a9a in term 1.
I20260806 23:55:02.397431 1829 leader_election.cc:302] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 322cfb9e5c5146cf8d296d858c4b6a9a, 8932d9a2a3d548a88298cbd1cd9f2476; no voters:
I20260806 23:55:02.397637 2036 raft_consensus.cc:2781] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:55:02.397893 2036 raft_consensus.cc:686] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 1 LEADER]: Becoming Leader. State: Replica: 322cfb9e5c5146cf8d296d858c4b6a9a, State: Running, Role: LEADER
I20260806 23:55:02.398273 2036 consensus_queue.cc:237] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:02.401888 1317 catalog_manager.cc:5495] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131). New cstate: current_term: 1 leader_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } health_report { overall_health: HEALTHY } } }
I20260806 23:55:02.423380 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.419970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.10.225.131:33835)
0806 23:55:02.420119 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:02.420124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:02.420138 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:55:02.420220 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:02.420222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:02.420276 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:02.420370 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:55:02.420619 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.421132 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.421963 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:02.422169 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:02.422255 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:02.422546 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:55:02.422991 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:55:02.423095 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:02.423168 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 23:55:02.423226 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:55:02.429903 1530 raft_consensus.cc:1261] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Refusing update from remote peer 322cfb9e5c5146cf8d296d858c4b6a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:55:02.430133 1723 raft_consensus.cc:1261] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 1 FOLLOWER]: Refusing update from remote peer 322cfb9e5c5146cf8d296d858c4b6a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:55:02.430735 2036 consensus_queue.cc:1035] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:55:02.430985 2038 consensus_queue.cc:1035] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:55:02.440263 2045 mvcc.cc:205] Tried to move back new op lower bound from 7315703817944760320 to 7315703817826398208. Current Snapshot: MvccSnapshot[applied={T|T < 7315703817944760320}]
I20260806 23:55:02.452395 1318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:55:02.452571 1318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:55:02.455257 1318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Sending DeleteTablet for 3 replicas of tablet 01b3d8dda71740cbbe5822686548afb1
I20260806 23:55:02.456182 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 1408
I20260806 23:55:02.456207 1698 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b3d8dda71740cbbe5822686548afb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:55:02 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:54618
I20260806 23:55:02.456189 1500 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b3d8dda71740cbbe5822686548afb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:55:02 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:59530
I20260806 23:55:02.456835 1889 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b3d8dda71740cbbe5822686548afb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:55:02 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:59680
I20260806 23:55:02.466359 2059 tablet_replica.cc:330] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: stopping tablet replica
I20260806 23:55:02.466784 2059 raft_consensus.cc:2229] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:55:02.467471 2059 raft_consensus.cc:2258] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:55:02.468385 2056 tablet_replica.cc:330] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: stopping tablet replica
I20260806 23:55:02.468753 2056 raft_consensus.cc:2229] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:55:02.469139 2056 raft_consensus.cc:2258] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:55:02.469401 2059 ts_tablet_manager.cc:1905] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:55:02.470676 2056 ts_tablet_manager.cc:1905] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:55:02.472417 2059 ts_tablet_manager.cc:1918] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:55:02.472528 2059 log.cc:1196] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/01b3d8dda71740cbbe5822686548afb1
I20260806 23:55:02.472862 2059 ts_tablet_manager.cc:1939] T 01b3d8dda71740cbbe5822686548afb1 P 322cfb9e5c5146cf8d296d858c4b6a9a: Deleting consensus metadata
I20260806 23:55:02.473820 2056 ts_tablet_manager.cc:1918] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:55:02.473989 2056 log.cc:1196] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/01b3d8dda71740cbbe5822686548afb1
I20260806 23:55:02.474352 2056 ts_tablet_manager.cc:1939] T 01b3d8dda71740cbbe5822686548afb1 P 8932d9a2a3d548a88298cbd1cd9f2476: Deleting consensus metadata
I20260806 23:55:02.474400 1301 catalog_manager.cc:4841] TS 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835): tablet 01b3d8dda71740cbbe5822686548afb1 (table test-table [id=4db65be700294381815f67acea0dd6d7]) successfully deleted
I20260806 23:55:02.475474 1301 catalog_manager.cc:4841] TS 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811): tablet 01b3d8dda71740cbbe5822686548afb1 (table test-table [id=4db65be700294381815f67acea0dd6d7]) successfully deleted
I20260806 23:55:02.477183 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 1607
W20260806 23:55:02.477185 1299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:39259 (error 108)
I20260806 23:55:02.485052 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:02.484696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:39259 (local address 127.0.0.1:59546)
0806 23:55:02.484882 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:02.484961 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 23:55:02.485245 1299 catalog_manager.cc:4584] TS 80c3d4caae9b4a3ea0a0f126770b9a34 (127.10.225.129:39259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01b3d8dda71740cbbe5822686548afb1: Network error: Client connection negotiation failed: client connection to 127.10.225.129:39259: connect: Connection refused (error 111)
I20260806 23:55:02.490399 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 1798
I20260806 23:55:02.509431 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 1274
I20260806 23:55:22.521840 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41925
--webserver_interface=127.10.225.190
--webserver_port=40659
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:41925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:55:22.641335 2061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:22.641639 2061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:22.641698 2061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:22.645839 2061 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 23:55:22.645910 2061 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 23:55:22.645932 2061 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 23:55:22.645949 2061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:55:22.645975 2061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:22.646009 2061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:22.646037 2061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:55:22.646066 2061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:55:22.651631 2061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:41925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:22.652845 2061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:22.653934 2061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:22.670011 2069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:22.670126 2061 server_base.cc:1034] running on GCE node
W20260806 23:55:22.670064 2072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:22.670064 2070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:22.671116 2061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:22.672483 2061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:22.673678 2061 hybrid_clock.cc:648] HybridClock initialized: now 1786060522673649 us; error 52 us; skew 500 ppm
Aug 06 23:55:22 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060522, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:22.681371 2061 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:55:22.682430 2061 webserver.cc:460] Webserver started at http://127.10.225.190:40659/ using document root <none> and password file <none>
I20260806 23:55:22.683033 2061 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:22.683111 2061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:22.688072 2061 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:55:22.690793 2082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:22.692428 2061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:55:22.692577 2061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "c736983ecca04e6780a5cad3743ec0a1"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "1ba90988054f97e29eaf788cded8e199"
server_key_iv: "f5292094dcaf5934024b2e8f9ff88c5e"
server_key_version: "encryptionkey@0"
I20260806 23:55:22.693217 2061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:22.706719 2061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:22.710258 2061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:22.710436 2061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:22.717947 2061 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:41925
I20260806 23:55:22.718372 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:41925 every 8 connection(s)
I20260806 23:55:22.719841 2061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:55:22.722198 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 2061
I20260806 23:55:22.722728 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:39259
--local_ip_for_outbound_sockets=127.10.225.129
--tserver_master_addrs=127.10.225.190:41925
--webserver_port=38163
--webserver_interface=127.10.225.129
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:55:22.726662 2166 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:55:22.729776 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Bootstrap starting.
I20260806 23:55:22.756534 2166 log.cc:824] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:22.768133 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:55:22.768641 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Bootstrap complete.
I20260806 23:55:22.774446 2166 raft_consensus.cc:348] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:22.774984 2166 raft_consensus.cc:729] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c736983ecca04e6780a5cad3743ec0a1, State: Initialized, Role: FOLLOWER
I20260806 23:55:22.775588 2166 consensus_queue.cc:260] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:22.775774 2166 raft_consensus.cc:388] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:55:22.775852 2166 raft_consensus.cc:482] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:55:22.775952 2166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 23:55:22.778626 2166 raft_consensus.cc:504] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:22.779016 2166 leader_election.cc:302] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c736983ecca04e6780a5cad3743ec0a1; no voters:
I20260806 23:55:22.779397 2166 leader_election.cc:288] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 23:55:22.779572 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 23:55:22.779871 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 2 LEADER]: Becoming Leader. State: Replica: c736983ecca04e6780a5cad3743ec0a1, State: Running, Role: LEADER
I20260806 23:55:22.780217 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:22.780776 2166 sys_catalog.cc:564] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:55:22.782820 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c736983ecca04e6780a5cad3743ec0a1. Latest consensus state: current_term: 2 leader_uuid: "c736983ecca04e6780a5cad3743ec0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } } }
I20260806 23:55:22.783027 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:55:22.784139 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c736983ecca04e6780a5cad3743ec0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } } }
I20260806 23:55:22.784286 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:55:22.787652 2193 catalog_manager.cc:1259] Loaded cluster ID: a7259a69b5b348d28ad419d91ecdb1e5
I20260806 23:55:22.787750 2193 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: loading cluster ID for follower catalog manager: success
I20260806 23:55:22.790422 2193 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: acquiring CA information for follower catalog manager: success
I20260806 23:55:22.792064 2193 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:55:22.792371 2192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:55:22.794827 2192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:55:22.795310 2192 catalog_manager.cc:1259] Loaded cluster ID: a7259a69b5b348d28ad419d91ecdb1e5
I20260806 23:55:22.795379 2192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:55:22.796110 2192 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:55:22.796609 2192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Loaded TSK: 0
I20260806 23:55:22.809871 2192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Generated new TSK 1
I20260806 23:55:22.810158 2192 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 23:55:22.873900 2170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:22.874171 2170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:22.874221 2170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:22.880046 2170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:22.880129 2170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:22.880201 2170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:55:22.886427 2170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:39259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=38163
--tserver_master_addrs=127.10.225.190:41925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:22.887552 2170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:22.888895 2170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:22.900177 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:22.900177 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:22.900564 2170 server_base.cc:1034] running on GCE node
W20260806 23:55:22.900177 2208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:22.901198 2170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:22.901893 2170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:22.903097 2170 hybrid_clock.cc:648] HybridClock initialized: now 1786060522903049 us; error 60 us; skew 500 ppm
Aug 06 23:55:22 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060522, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:22.905851 2170 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:55:22.906822 2170 webserver.cc:460] Webserver started at http://127.10.225.129:38163/ using document root <none> and password file <none>
I20260806 23:55:22.907397 2170 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:22.907473 2170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:22.911725 2170 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:55:22.913932 2217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:22.915218 2170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 23:55:22.915321 2170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "80c3d4caae9b4a3ea0a0f126770b9a34"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "b7d8d57b098b0162662b3e3f70c56990"
server_key_iv: "735f47d3e71e29ab7555c8abb2a99060"
server_key_version: "encryptionkey@0"
I20260806 23:55:22.915946 2170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:22.931491 2170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:22.934685 2170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:22.934849 2170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:22.935423 2170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:55:22.936611 2224 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 23:55:22.941459 2170 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 23:55:22.941541 2170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 23:55:22.941601 2170 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 23:55:22.943354 2170 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 23:55:22.943403 2170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 23:55:22.943742 2224 tablet_bootstrap.cc:492] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Bootstrap starting.
I20260806 23:55:22.954247 2170 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:39259
I20260806 23:55:22.954283 2387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:39259 every 8 connection(s)
I20260806 23:55:22.955279 2170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:55:22.964270 2224 log.cc:824] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:22.964334 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 2170
I20260806 23:55:22.964859 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:38811
--local_ip_for_outbound_sockets=127.10.225.130
--tserver_master_addrs=127.10.225.190:41925
--webserver_port=39135
--webserver_interface=127.10.225.130
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:55:22 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060522, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:55:22.973033 2224 tablet_bootstrap.cc:492] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:55:22.973484 2224 tablet_bootstrap.cc:492] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Bootstrap complete.
I20260806 23:55:22.974819 2224 ts_tablet_manager.cc:1397] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260806 23:55:22.975059 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:22.957237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:44461)
0806 23:55:22.957912 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:22.957966 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:22.958761 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:22.960233 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:22.960240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:22.960603 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:55:22.961109 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:22.961123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:22.963413 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:22.963416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:22.964392 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:22.964406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:22.965439 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:22.965453 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:22.965797 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:22.966720 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:55:22.966742 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:55:22.968279 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:55:22.971087 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:22.971093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:22.971105 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:22.971467 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:22.972000 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:22.972004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:22.972006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:22.972183 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:22.973247 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:55:22.973255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:55:22.973561 (+ 306us) client_negotiation.cc:769] Sending connection context
0806 23:55:22.973762 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 23:55:22.974104 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":170,"threads_started":1}
I20260806 23:55:22.975633 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:22.957374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:44461 (local address 127.10.225.190:41925)
0806 23:55:22.957776 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:22.957799 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:22.958103 (+ 304us) server_negotiation.cc:407] Connection header received
0806 23:55:22.959254 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:22.959270 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:22.959760 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:22.960152 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:22.961366 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:22.963227 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:22.964545 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:22.965184 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:22.965357 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:22.968486 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:22.968491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:22.968504 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:22.968533 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:22.970905 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:22.971635 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:22.971644 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:22.971652 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:22.971881 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:22.972297 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:22.972301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:22.972302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:22.973062 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:22.973257 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:22.973926 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 23:55:22.974344 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
I20260806 23:55:22.977605 2224 raft_consensus.cc:348] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:22.978116 2224 raft_consensus.cc:729] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c3d4caae9b4a3ea0a0f126770b9a34, State: Initialized, Role: FOLLOWER
I20260806 23:55:22.978451 2389 heartbeater.cc:347] Connected to a master server at 127.10.225.190:41925
I20260806 23:55:22.978677 2389 heartbeater.cc:464] Registering TS with master...
I20260806 23:55:22.978606 2224 consensus_queue.cc:260] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } } peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } }
I20260806 23:55:22.979271 2389 heartbeater.cc:511] Master 127.10.225.190:41925 requested a full tablet report, sending...
I20260806 23:55:22.979697 2224 ts_tablet_manager.cc:1428] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260806 23:55:22.982049 2110 ts_manager.cc:194] Registered new tserver with Master: 80c3d4caae9b4a3ea0a0f126770b9a34 (127.10.225.129:39259)
I20260806 23:55:22.985469 2110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:44461
I20260806 23:55:22.988206 2389 heartbeater.cc:503] Master 127.10.225.190:41925 was elected leader, sending a full tablet report...
Aug 06 23:55:22 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060522, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.129@KRBTEST.COM
I20260806 23:55:22.996812 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:22.985165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:39259 (local address 127.0.0.1:33518)
0806 23:55:22.985789 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:22.985836 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:22.986003 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:22.986753 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:22.986760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:22.987207 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:55:22.987361 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:22.987369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:22.988663 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:22.988667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:22.989497 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:22.989517 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:22.990478 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:22.990483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:22.990592 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:22.991160 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:55:22.991172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:55:22.992864 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:55:22.994956 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:22.994964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:22.994969 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:22.995193 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:22.995535 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:22.995538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:22.995540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:22.995631 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:22.996226 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:55:22.996232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:55:22.996361 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 23:55:22.996480 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 23:55:22.996651 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":184,"threads_started":1}
I20260806 23:55:22.996917 2398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:22.985194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.10.225.129:39259)
0806 23:55:22.986281 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:22.986293 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:22.986318 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:55:22.986404 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:22.986410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:22.986575 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:22.986712 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:22.987500 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:22.988548 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:22.989707 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:22.990334 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:22.990447 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:22.993018 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:22.993029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:22.993043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:22.993070 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:22.994830 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:22.995302 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:22.995308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:22.995310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:22.995404 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:22.995770 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:22.995773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:22.995775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:22.996102 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:22.996244 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:22.996573 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 23:55:22.996752 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":991,"thread_start_us":160,"threads_started":1}
I20260806 23:55:22.998154 2287 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b3d8dda71740cbbe5822686548afb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:55:02 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:33518
I20260806 23:55:22.999099 2401 tablet_replica.cc:330] stopping tablet replica
I20260806 23:55:22.999388 2401 raft_consensus.cc:2229] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:55:22.999626 2401 raft_consensus.cc:2258] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:55:23.000595 2401 ts_tablet_manager.cc:1905] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:55:23.004132 2401 ts_tablet_manager.cc:1918] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:55:23.004215 2401 log.cc:1196] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal/wals/01b3d8dda71740cbbe5822686548afb1
I20260806 23:55:23.004441 2401 ts_tablet_manager.cc:1939] T 01b3d8dda71740cbbe5822686548afb1 P 80c3d4caae9b4a3ea0a0f126770b9a34: Deleting consensus metadata
I20260806 23:55:23.006091 2085 catalog_manager.cc:4841] TS 80c3d4caae9b4a3ea0a0f126770b9a34 (127.10.225.129:39259): tablet 01b3d8dda71740cbbe5822686548afb1 (table test-table [id=4db65be700294381815f67acea0dd6d7]) successfully deleted
W20260806 23:55:23.093657 2396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:23.093986 2396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:23.094051 2396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:23.098577 2396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:23.098649 2396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:23.098737 2396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:55:23.105014 2396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:38811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=39135
--tserver_master_addrs=127.10.225.190:41925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:23.106177 2396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:23.107266 2396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:55:23.117336 2396 server_base.cc:1034] running on GCE node
W20260806 23:55:23.117295 2414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:23.117285 2418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:23.117283 2416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:23.117887 2396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:23.118548 2396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:23.119755 2396 hybrid_clock.cc:648] HybridClock initialized: now 1786060523119730 us; error 43 us; skew 500 ppm
Aug 06 23:55:23 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:23.123302 2396 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:55:23.124589 2396 webserver.cc:460] Webserver started at http://127.10.225.130:39135/ using document root <none> and password file <none>
I20260806 23:55:23.125177 2396 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:23.125260 2396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:23.129516 2396 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:55:23.132011 2429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:23.133503 2396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:55:23.133626 2396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "8932d9a2a3d548a88298cbd1cd9f2476"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "208f4efd8498880f90ff784dc7377fa7"
server_key_iv: "d7c732eef9c3b98095d8b0700c822bca"
server_key_version: "encryptionkey@0"
I20260806 23:55:23.134199 2396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:23.156889 2396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:23.161095 2396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:23.161341 2396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:23.162160 2396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:55:23.163548 2396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:55:23.163633 2396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:23.163722 2396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:55:23.163759 2396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:23.173751 2396 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:38811
I20260806 23:55:23.173899 2599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:38811 every 8 connection(s)
I20260806 23:55:23.174878 2396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:55:23.175285 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 2396
I20260806 23:55:23.175778 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:33835
--local_ip_for_outbound_sockets=127.10.225.131
--tserver_master_addrs=127.10.225.190:41925
--webserver_port=39625
--webserver_interface=127.10.225.131
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 23:55:23 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:55:23.189388 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.176961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:51753 (local address 127.10.225.190:41925)
0806 23:55:23.177122 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.177127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.178053 (+ 926us) server_negotiation.cc:407] Connection header received
0806 23:55:23.179474 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.179477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.179541 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.179619 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:23.180957 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.181507 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.182781 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.183138 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.183256 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.185833 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:23.185836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:23.185838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:23.185871 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:23.187235 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:23.187770 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:23.187773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:23.187775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:23.187851 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:23.188191 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:23.188193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:23.188195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:23.188389 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:23.188474 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:23.188998 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 23:55:23.189136 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:55:23.190255 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.177380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:51753)
0806 23:55:23.177877 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:23.177931 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:23.179257 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:23.179831 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:23.179838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:23.180219 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:55:23.180809 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.180824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.181633 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.181635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.182667 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.182678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.183333 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.183338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.183522 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.184128 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:55:23.184145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:55:23.185653 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:55:23.187348 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:23.187354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:23.187381 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:23.187643 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:23.187944 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:23.187948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:23.187950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:23.188093 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:23.188488 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:55:23.188495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:55:23.188755 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 23:55:23.188923 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 23:55:23.189268 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":131,"threads_started":1}
I20260806 23:55:23.191694 2601 heartbeater.cc:347] Connected to a master server at 127.10.225.190:41925
I20260806 23:55:23.191954 2601 heartbeater.cc:464] Registering TS with master...
I20260806 23:55:23.192467 2601 heartbeater.cc:511] Master 127.10.225.190:41925 requested a full tablet report, sending...
I20260806 23:55:23.193671 2110 ts_manager.cc:194] Registered new tserver with Master: 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811)
I20260806 23:55:23.194260 2110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:51753
W20260806 23:55:23.298237 2603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:23.298549 2603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:23.298610 2603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:23.302891 2603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:23.302958 2603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:23.303035 2603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:55:23.308563 2603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:33835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=39625
--tserver_master_addrs=127.10.225.190:41925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:23.309711 2603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:23.310762 2603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:23.322966 2610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:23.322994 2609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:23.323217 2603 server_base.cc:1034] running on GCE node
W20260806 23:55:23.322966 2614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:23.323782 2603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:23.324481 2603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:23.325655 2603 hybrid_clock.cc:648] HybridClock initialized: now 1786060523325598 us; error 68 us; skew 500 ppm
Aug 06 23:55:23 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:23.328433 2603 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:55:23.329337 2603 webserver.cc:460] Webserver started at http://127.10.225.131:39625/ using document root <none> and password file <none>
I20260806 23:55:23.329916 2603 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:23.330004 2603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:23.334200 2603 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 23:55:23.336510 2624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:23.337813 2603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:55:23.337929 2603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "322cfb9e5c5146cf8d296d858c4b6a9a"
format_stamp: "Formatted at 2026-08-06 23:55:02 on dist-test-slave-nj21"
server_key: "eddf4fc75d1798373fef18e4824d1c04"
server_key_iv: "2d7735e7142741d788b8d55ea93e5777"
server_key_version: "encryptionkey@0"
I20260806 23:55:23.338510 2603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:23.368048 2603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:23.371232 2603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:23.371424 2603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:23.372094 2603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:55:23.372949 2603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:55:23.372997 2603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:23.373052 2603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:55:23.373076 2603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:23.383437 2603 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:33835
I20260806 23:55:23.383505 2806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:33835 every 8 connection(s)
I20260806 23:55:23.384514 2603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
Aug 06 23:55:23 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:55:23.394483 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 2603
I20260806 23:55:23.398602 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.386399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:50751 (local address 127.10.225.190:41925)
0806 23:55:23.386569 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.386574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.387591 (+ 1017us) server_negotiation.cc:407] Connection header received
0806 23:55:23.388957 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.388960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.389011 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.389087 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:23.390354 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.390879 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.392064 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.392302 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.392395 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.394916 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:23.394920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:23.394922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:23.394955 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:23.396416 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:23.396940 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:23.396944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:23.396945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:23.397024 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:23.397388 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:23.397391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:23.397393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:23.397662 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:23.397749 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:23.398294 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 23:55:23.398432 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 23:55:23.399842 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.386916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:50751)
0806 23:55:23.387430 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:23.387481 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:23.388744 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:23.389240 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:23.389246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:23.389644 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:55:23.390156 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.390171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.391011 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.391014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.391931 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.391942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.392479 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.392482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.392643 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.393186 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:55:23.393202 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:55:23.394740 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:55:23.396531 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:23.396538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:23.396558 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:23.396833 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:23.397112 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:23.397115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:23.397117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:23.397292 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:23.397764 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:55:23.397771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:55:23.398046 (+ 275us) client_negotiation.cc:769] Sending connection context
0806 23:55:23.398233 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 23:55:23.398696 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260806 23:55:23.401139 2807 heartbeater.cc:347] Connected to a master server at 127.10.225.190:41925
I20260806 23:55:23.401392 2807 heartbeater.cc:464] Registering TS with master...
I20260806 23:55:23.401989 2807 heartbeater.cc:511] Master 127.10.225.190:41925 requested a full tablet report, sending...
I20260806 23:55:23.403098 2110 ts_manager.cc:194] Registered new tserver with Master: 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835)
I20260806 23:55:23.403863 2110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:50751
I20260806 23:55:23.404580 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.395176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.10.225.190:41925)
0806 23:55:23.395481 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.395485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.395624 (+ 139us) server_negotiation.cc:407] Connection header received
0806 23:55:23.395773 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.395776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.395821 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.395912 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:23.397231 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.397941 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.398981 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.399358 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.399479 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.401270 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:23.401274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:23.401277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:23.401302 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:23.402661 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:23.403145 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:23.403147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:23.403149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:23.403245 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:23.403659 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:23.403662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:23.403664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:23.403892 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:23.404092 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:23.404302 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 23:55:23.404431 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260806 23:55:23.406221 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:55:23.410178 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.406920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.10.225.190:41925)
0806 23:55:23.407028 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.407031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.407248 (+ 217us) server_negotiation.cc:407] Connection header received
0806 23:55:23.407366 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.407368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.407468 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.407545 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:55:23.407773 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.408290 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.409096 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.409314 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.409391 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.409736 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:55:23.409867 (+ 131us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 23:55:23.410084 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58098: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 23:55:23.410247 2816 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58098: token expired
W20260806 23:55:23.410665 1993 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.225.190:41925)
I20260806 23:55:23.418908 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.411016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.10.225.190:41925)
0806 23:55:23.411116 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.411118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.411130 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:55:23.411207 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.411209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.411244 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.411306 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:23.412220 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.412684 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.413508 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.413739 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.413817 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.415112 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:23.415115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:23.415117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:23.415139 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:23.416944 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:23.417404 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:23.417409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:23.417410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:23.417568 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:23.417994 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:23.418004 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:23.418005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:23.418214 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:23.418348 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:23.418657 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 23:55:23.418773 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 23:55:23.421864 2110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:55:23.425830 2110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:55:23.434547 2287 tablet_service.cc:1467] Processing CreateTablet for tablet c20ffabc0fd947f98704cc9290614e71 (DEFAULT_TABLE table=test-table [id=9d1d318665854e83845556a69ad3c1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:55:23.435210 2287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20ffabc0fd947f98704cc9290614e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:55:23.437942 2224 tablet_bootstrap.cc:492] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Bootstrap starting.
I20260806 23:55:23.438797 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.433489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:38811 (local address 127.0.0.1:53210)
0806 23:55:23.434003 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:23.434020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:23.434104 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:23.434683 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:23.434686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:23.434703 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:55:23.434821 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.434828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.436070 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.436073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.436897 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.436906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.438367 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.438371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.438484 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.438530 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 23:55:23.438574 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 23:55:23.438649 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442}
I20260806 23:55:23.438886 2224 tablet_bootstrap.cc:654] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Neither blocks nor log segments found. Creating new log.
I20260806 23:55:23.439638 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.433652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.10.225.130:38811)
0806 23:55:23.434179 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.434192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.434217 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:55:23.434307 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.434313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.434539 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.434690 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:55:23.434913 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.435964 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.437023 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.438223 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.439006 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.439169 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:23.439226 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 23:55:23.439429 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":173,"threads_started":1}
I20260806 23:55:23.440450 2224 tablet_bootstrap.cc:492] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: No bootstrap required, opened a new log
I20260806 23:55:23.440538 2224 ts_tablet_manager.cc:1397] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 23:55:23.440976 2224 raft_consensus.cc:348] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:23.441102 2224 raft_consensus.cc:374] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:55:23.441140 2224 raft_consensus.cc:729] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c3d4caae9b4a3ea0a0f126770b9a34, State: Initialized, Role: FOLLOWER
I20260806 23:55:23.441439 2224 consensus_queue.cc:260] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:23.442158 2224 ts_tablet_manager.cc:1428] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 23:55:23.443642 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.433725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:33835 (local address 127.0.0.1:42212)
0806 23:55:23.434277 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:23.434288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:23.434357 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:23.439016 (+ 4659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:23.439019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:23.439040 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:55:23.439136 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.439142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.441135 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.441139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.442159 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.442171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.443057 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.443061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.443151 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.443175 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:55:23.443256 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 23:55:23.443465 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":105,"threads_started":1}
I20260806 23:55:23.443473 2503 tablet_service.cc:1467] Processing CreateTablet for tablet c20ffabc0fd947f98704cc9290614e71 (DEFAULT_TABLE table=test-table [id=9d1d318665854e83845556a69ad3c1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:55:23.444576 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.433896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.10.225.131:33835)
0806 23:55:23.438283 (+ 4387us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.438296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.438322 (+ 26us) server_negotiation.cc:407] Connection header received
0806 23:55:23.438418 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.438424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.438648 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.438817 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:55:23.439257 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.441009 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.442317 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.442933 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.443052 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.443181 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:23.443989 (+ 808us) server_negotiation.cc:299] Negotiation successful
0806 23:55:23.444426 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4267,"thread_start_us":114,"threads_started":1}
I20260806 23:55:23.444674 2503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20ffabc0fd947f98704cc9290614e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:55:23.447851 2700 tablet_service.cc:1467] Processing CreateTablet for tablet c20ffabc0fd947f98704cc9290614e71 (DEFAULT_TABLE table=test-table [id=9d1d318665854e83845556a69ad3c1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:55:23.449270 2700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20ffabc0fd947f98704cc9290614e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:55:23.454062 2825 tablet_bootstrap.cc:492] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: Bootstrap starting.
W20260806 23:55:23.456530 2390 tablet.cc:2369] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:55:23.456892 2825 tablet_bootstrap.cc:654] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: Neither blocks nor log segments found. Creating new log.
I20260806 23:55:23.457166 2826 tablet_bootstrap.cc:492] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: Bootstrap starting.
I20260806 23:55:23.457801 2825 log.cc:824] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:23.459347 2826 tablet_bootstrap.cc:654] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: Neither blocks nor log segments found. Creating new log.
I20260806 23:55:23.460086 2825 tablet_bootstrap.cc:492] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: No bootstrap required, opened a new log
I20260806 23:55:23.460326 2825 ts_tablet_manager.cc:1397] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:55:23.460852 2826 log.cc:824] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:23.462904 2826 tablet_bootstrap.cc:492] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: No bootstrap required, opened a new log
I20260806 23:55:23.463109 2826 ts_tablet_manager.cc:1397] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:55:23.464155 2825 raft_consensus.cc:348] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:23.464448 2825 raft_consensus.cc:374] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:55:23.464504 2825 raft_consensus.cc:729] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8932d9a2a3d548a88298cbd1cd9f2476, State: Initialized, Role: FOLLOWER
I20260806 23:55:23.464985 2825 consensus_queue.cc:260] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:23.465705 2601 heartbeater.cc:503] Master 127.10.225.190:41925 was elected leader, sending a full tablet report...
I20260806 23:55:23.466081 2825 ts_tablet_manager.cc:1428] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 23:55:23.466831 2826 raft_consensus.cc:348] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:23.467096 2826 raft_consensus.cc:374] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:55:23.467147 2826 raft_consensus.cc:729] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322cfb9e5c5146cf8d296d858c4b6a9a, State: Initialized, Role: FOLLOWER
I20260806 23:55:23.467717 2826 consensus_queue.cc:260] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:23.468497 2807 heartbeater.cc:503] Master 127.10.225.190:41925 was elected leader, sending a full tablet report...
I20260806 23:55:23.468696 2826 ts_tablet_manager.cc:1428] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 23:55:23.529304 2830 raft_consensus.cc:482] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:55:23.529556 2830 raft_consensus.cc:504] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:23.530813 2830 leader_election.cc:288] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835), 80c3d4caae9b4a3ea0a0f126770b9a34 (127.10.225.129:39259)
I20260806 23:55:23.535018 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.531325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:57423 (local address 127.10.225.129:39259)
0806 23:55:23.531592 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.531596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.531611 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:55:23.531707 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.531711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.531783 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.531891 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:55:23.532182 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.532767 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.533956 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.534490 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.534598 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.534757 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:23.534811 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 23:55:23.534883 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":104,"threads_started":1}
I20260806 23:55:23.535017 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.531113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:39259 (local address 127.10.225.130:57423)
0806 23:55:23.531335 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:23.531350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:23.531449 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:23.531902 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:23.531905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:23.531921 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:55:23.532070 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.532078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.532881 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.532885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.533829 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.533839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.534598 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.534602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.534704 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.534727 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:55:23.534787 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 23:55:23.534864 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":92}
I20260806 23:55:23.535216 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.531303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:41651 (local address 127.10.225.131:33835)
0806 23:55:23.531451 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.531456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.531753 (+ 297us) server_negotiation.cc:407] Connection header received
0806 23:55:23.531851 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.531854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.531906 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.531980 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:55:23.532241 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.532767 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.533968 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.534516 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.534641 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.534695 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:23.535000 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 23:55:23.535081 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 23:55:23.535631 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.531130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:33835 (local address 127.10.225.130:41651)
0806 23:55:23.531618 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:23.531631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:23.531764 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:23.531992 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:23.531995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:23.532010 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:55:23.532111 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.532116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.532881 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.532885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.533828 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:23.533863 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.534758 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.534761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:23.534859 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.534882 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:55:23.535422 (+ 540us) client_negotiation.cc:240] Negotiation successful
0806 23:55:23.535498 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":74,"thread_start_us":122,"threads_started":1}
I20260806 23:55:23.535714 2314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ffabc0fd947f98704cc9290614e71" candidate_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" is_pre_election: true
I20260806 23:55:23.536070 2314 raft_consensus.cc:2454] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8932d9a2a3d548a88298cbd1cd9f2476 in term 0.
I20260806 23:55:23.536233 2736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ffabc0fd947f98704cc9290614e71" candidate_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" is_pre_election: true
I20260806 23:55:23.536502 2736 raft_consensus.cc:2454] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8932d9a2a3d548a88298cbd1cd9f2476 in term 0.
I20260806 23:55:23.536623 2432 leader_election.cc:302] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c3d4caae9b4a3ea0a0f126770b9a34, 8932d9a2a3d548a88298cbd1cd9f2476; no voters:
I20260806 23:55:23.536896 2830 raft_consensus.cc:2781] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:55:23.536971 2830 raft_consensus.cc:482] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:55:23.537002 2830 raft_consensus.cc:3037] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:55:23.538074 2830 raft_consensus.cc:504] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:23.538427 2830 leader_election.cc:288] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [CANDIDATE]: Term 1 election: Requested vote from peers 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835), 80c3d4caae9b4a3ea0a0f126770b9a34 (127.10.225.129:39259)
I20260806 23:55:23.539041 2736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ffabc0fd947f98704cc9290614e71" candidate_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a"
I20260806 23:55:23.539183 2736 raft_consensus.cc:3037] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:55:23.539183 2314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ffabc0fd947f98704cc9290614e71" candidate_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34"
I20260806 23:55:23.539424 2314 raft_consensus.cc:3037] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:55:23.540534 2736 raft_consensus.cc:2454] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8932d9a2a3d548a88298cbd1cd9f2476 in term 1.
I20260806 23:55:23.540985 2434 leader_election.cc:302] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 322cfb9e5c5146cf8d296d858c4b6a9a, 8932d9a2a3d548a88298cbd1cd9f2476; no voters:
I20260806 23:55:23.541172 2830 raft_consensus.cc:2781] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:55:23.541399 2830 raft_consensus.cc:686] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 1 LEADER]: Becoming Leader. State: Replica: 8932d9a2a3d548a88298cbd1cd9f2476, State: Running, Role: LEADER
I20260806 23:55:23.541389 2314 raft_consensus.cc:2454] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8932d9a2a3d548a88298cbd1cd9f2476 in term 1.
I20260806 23:55:23.541812 2830 consensus_queue.cc:237] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:23.545123 2110 catalog_manager.cc:5495] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130). New cstate: current_term: 1 leader_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } health_report { overall_health: UNKNOWN } } }
I20260806 23:55:23.567806 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.564446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.10.225.130:38811)
0806 23:55:23.564574 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:23.564578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:23.564592 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:55:23.564692 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:23.564694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:23.564749 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:23.564835 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:55:23.565004 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.565494 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.566371 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:23.566589 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:23.566680 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:23.566978 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:55:23.567405 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:55:23.567499 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:23.567568 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 23:55:23.567624 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:55:23.574843 2314 raft_consensus.cc:1261] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Refusing update from remote peer 8932d9a2a3d548a88298cbd1cd9f2476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:55:23.574846 2736 raft_consensus.cc:1261] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 1 FOLLOWER]: Refusing update from remote peer 8932d9a2a3d548a88298cbd1cd9f2476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:55:23.575717 2830 consensus_queue.cc:1035] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:55:23.575937 2840 consensus_queue.cc:1035] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:55:23.583719 2846 mvcc.cc:205] Tried to move back new op lower bound from 7315703904553271296 to 7315703904430342144. Current Snapshot: MvccSnapshot[applied={T|T < 7315703904553271296}]
I20260806 23:55:23.596681 2101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:55:23.596936 2101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:55:23.600270 2101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Sending DeleteTablet for 3 replicas of tablet c20ffabc0fd947f98704cc9290614e71
I20260806 23:55:23.601311 2287 tablet_service.cc:1514] Processing DeleteTablet for tablet c20ffabc0fd947f98704cc9290614e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:55:23 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:33518
I20260806 23:55:23.601383 2503 tablet_service.cc:1514] Processing DeleteTablet for tablet c20ffabc0fd947f98704cc9290614e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:55:23 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:53210
I20260806 23:55:23.601506 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 2170
I20260806 23:55:23.602579 2700 tablet_service.cc:1514] Processing DeleteTablet for tablet c20ffabc0fd947f98704cc9290614e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:55:23 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:42212
I20260806 23:55:23.606062 2862 tablet_replica.cc:330] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: stopping tablet replica
I20260806 23:55:23.606422 2862 raft_consensus.cc:2229] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:55:23.606875 2862 raft_consensus.cc:2258] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:55:23.611143 2862 ts_tablet_manager.cc:1905] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:55:23.613760 2863 tablet_replica.cc:330] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: stopping tablet replica
I20260806 23:55:23.614156 2863 raft_consensus.cc:2229] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:55:23.614424 2862 ts_tablet_manager.cc:1918] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:55:23.614557 2863 raft_consensus.cc:2258] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:55:23.614573 2862 log.cc:1196] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/c20ffabc0fd947f98704cc9290614e71
I20260806 23:55:23.615594 2862 ts_tablet_manager.cc:1939] T c20ffabc0fd947f98704cc9290614e71 P 8932d9a2a3d548a88298cbd1cd9f2476: Deleting consensus metadata
I20260806 23:55:23.616266 2863 ts_tablet_manager.cc:1905] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:55:23.617190 2089 catalog_manager.cc:4841] TS 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811): tablet c20ffabc0fd947f98704cc9290614e71 (table test-table [id=9d1d318665854e83845556a69ad3c1b0]) successfully deleted
I20260806 23:55:23.619028 2863 ts_tablet_manager.cc:1918] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:55:23.619130 2863 log.cc:1196] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/c20ffabc0fd947f98704cc9290614e71
I20260806 23:55:23.619576 2863 ts_tablet_manager.cc:1939] T c20ffabc0fd947f98704cc9290614e71 P 322cfb9e5c5146cf8d296d858c4b6a9a: Deleting consensus metadata
I20260806 23:55:23.620986 2089 catalog_manager.cc:4841] TS 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835): tablet c20ffabc0fd947f98704cc9290614e71 (table test-table [id=9d1d318665854e83845556a69ad3c1b0]) successfully deleted
W20260806 23:55:23.621192 2085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:39259 (error 108)
I20260806 23:55:23.622195 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:23.621915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:39259 (local address 127.0.0.1:33524)
0806 23:55:23.622038 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:23.622105 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 23:55:23.622314 2085 catalog_manager.cc:4584] TS 80c3d4caae9b4a3ea0a0f126770b9a34 (127.10.225.129:39259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c20ffabc0fd947f98704cc9290614e71: Network error: Client connection negotiation failed: client connection to 127.10.225.129:39259: connect: Connection refused (error 111)
I20260806 23:55:23.623035 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 2396
I20260806 23:55:23.634177 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 2603
I20260806 23:55:23.649226 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 2061
I20260806 23:55:43.665323 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41925
--webserver_interface=127.10.225.190
--webserver_port=40659
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:41925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:55:43.785130 2866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:43.785531 2866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:43.785637 2866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:43.790120 2866 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 23:55:43.790196 2866 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 23:55:43.790211 2866 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 23:55:43.790225 2866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:55:43.790242 2866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:43.790261 2866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:43.790295 2866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:55:43.790324 2866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:55:43.796293 2866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:41925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:43.797473 2866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:43.798550 2866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:43.807798 2874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:43.807826 2873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:43.807794 2877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:43.807857 2866 server_base.cc:1034] running on GCE node
I20260806 23:55:43.808609 2866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:43.809731 2866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:43.810940 2866 hybrid_clock.cc:648] HybridClock initialized: now 1786060543810923 us; error 39 us; skew 500 ppm
Aug 06 23:55:43 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060543, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:43.813874 2866 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:55:43.814790 2866 webserver.cc:460] Webserver started at http://127.10.225.190:40659/ using document root <none> and password file <none>
I20260806 23:55:43.815325 2866 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:43.815387 2866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:43.819226 2866 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:55:43.822193 2888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:43.824141 2866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:55:43.824329 2866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "c736983ecca04e6780a5cad3743ec0a1"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "1ba90988054f97e29eaf788cded8e199"
server_key_iv: "f5292094dcaf5934024b2e8f9ff88c5e"
server_key_version: "encryptionkey@0"
I20260806 23:55:43.825114 2866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:43.847926 2866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:43.851356 2866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:43.851521 2866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:43.858994 2866 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:41925
I20260806 23:55:43.859294 2962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:41925 every 8 connection(s)
I20260806 23:55:43.860460 2866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:55:43.863663 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 2866
I20260806 23:55:43.864399 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:39259
--local_ip_for_outbound_sockets=127.10.225.129
--tserver_master_addrs=127.10.225.190:41925
--webserver_port=38163
--webserver_interface=127.10.225.129
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:55:43.865144 2963 sys_catalog.cc:263] Verifying existing consensus state
I20260806 23:55:43.866851 2963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Bootstrap starting.
I20260806 23:55:43.891019 2963 log.cc:824] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:43.904946 2963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:55:43.905341 2963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Bootstrap complete.
I20260806 23:55:43.909442 2963 raft_consensus.cc:348] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:43.909873 2963 raft_consensus.cc:729] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c736983ecca04e6780a5cad3743ec0a1, State: Initialized, Role: FOLLOWER
I20260806 23:55:43.910383 2963 consensus_queue.cc:260] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:43.910517 2963 raft_consensus.cc:388] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:55:43.910591 2963 raft_consensus.cc:482] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:55:43.910691 2963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 23:55:43.912693 2963 raft_consensus.cc:504] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:43.913025 2963 leader_election.cc:302] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c736983ecca04e6780a5cad3743ec0a1; no voters:
I20260806 23:55:43.913339 2963 leader_election.cc:288] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 23:55:43.913462 2969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 23:55:43.913666 2969 raft_consensus.cc:686] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [term 3 LEADER]: Becoming Leader. State: Replica: c736983ecca04e6780a5cad3743ec0a1, State: Running, Role: LEADER
I20260806 23:55:43.914029 2969 consensus_queue.cc:237] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } }
I20260806 23:55:43.914798 2963 sys_catalog.cc:564] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:55:43.915230 2969 sys_catalog.cc:455] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c736983ecca04e6780a5cad3743ec0a1. Latest consensus state: current_term: 3 leader_uuid: "c736983ecca04e6780a5cad3743ec0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } } }
I20260806 23:55:43.915352 2969 sys_catalog.cc:458] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:55:43.916108 2970 sys_catalog.cc:455] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c736983ecca04e6780a5cad3743ec0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736983ecca04e6780a5cad3743ec0a1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41925 } } }
I20260806 23:55:43.916275 2970 sys_catalog.cc:458] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:55:43.916875 2973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:55:43.922523 2973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:55:43.923259 2973 catalog_manager.cc:1259] Loaded cluster ID: a7259a69b5b348d28ad419d91ecdb1e5
I20260806 23:55:43.923537 2973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:55:43.926378 2973 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:55:43.927145 2973 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Loaded TSK: 0
I20260806 23:55:43.927232 2973 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c736983ecca04e6780a5cad3743ec0a1: Loaded TSK: 1
F20260806 23:55:43.928366 2973 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f559261c9ad google::LogMessage::Fail() at ??:0
@ 0x7f559261f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f559261c4de google::LogMessage::Flush() at ??:0
@ 0x7f559261da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f55952e018d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f55952d8f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f559cf857bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f559cf5f99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f559cf60178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f559cfa1df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f559cfc082b std::function<>::operator()() at ??:0
@ 0x7f559cf5fb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f559cf60f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f559cf5d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f559cfa1640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f559486e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f559486ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f55948703f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f559485f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5595ea36db start_thread at ??:0
@ 0x7f559111a61f clone at ??:0
W20260806 23:55:44.015697 2966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:44.015969 2966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:44.016026 2966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:44.020277 2966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:44.020331 2966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:44.020393 2966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:55:44.027343 2966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:39259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=38163
--tserver_master_addrs=127.10.225.190:41925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:44.028657 2966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:44.029742 2966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:44.039541 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:44.039543 3036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:44.039541 3034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:44.039924 2966 server_base.cc:1034] running on GCE node
I20260806 23:55:44.040421 2966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:44.041229 2966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:44.042424 2966 hybrid_clock.cc:648] HybridClock initialized: now 1786060544042396 us; error 38 us; skew 500 ppm
Aug 06 23:55:44 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:44.046550 2966 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:55:44.047753 2966 webserver.cc:460] Webserver started at http://127.10.225.129:38163/ using document root <none> and password file <none>
I20260806 23:55:44.048494 2966 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:44.048589 2966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:44.054209 2966 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 23:55:44.057070 3052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:44.058696 2966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:55:44.058840 2966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "80c3d4caae9b4a3ea0a0f126770b9a34"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "b7d8d57b098b0162662b3e3f70c56990"
server_key_iv: "735f47d3e71e29ab7555c8abb2a99060"
server_key_version: "encryptionkey@0"
I20260806 23:55:44.059589 2966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:44.084987 2966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:44.089607 2966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:44.089838 2966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:44.090669 2966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:55:44.092154 3061 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 23:55:44.096580 2966 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 23:55:44.096669 2966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 23:55:44.096743 2966 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 23:55:44.099103 2966 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 23:55:44.099169 2966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 23:55:44.099484 3061 tablet_bootstrap.cc:492] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Bootstrap starting.
I20260806 23:55:44.110832 2966 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:39259
I20260806 23:55:44.111137 3213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:39259 every 8 connection(s)
I20260806 23:55:44.112192 2966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:55:44.114166 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 2966
I20260806 23:55:44.114670 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:38811
--local_ip_for_outbound_sockets=127.10.225.130
--tserver_master_addrs=127.10.225.190:41925
--webserver_port=39135
--webserver_interface=127.10.225.130
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:55:44.122377 3061 log.cc:824] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Log is configured to *not* fsync() on all Append() calls
I20260806 23:55:44.133044 3061 tablet_bootstrap.cc:492] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 23:55:44.133490 3061 tablet_bootstrap.cc:492] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Bootstrap complete.
I20260806 23:55:44.134919 3061 ts_tablet_manager.cc:1397] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Time spent bootstrapping tablet: real 0.036s user 0.019s sys 0.011s
I20260806 23:55:44.138551 3061 raft_consensus.cc:348] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:44.139158 3061 raft_consensus.cc:729] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c3d4caae9b4a3ea0a0f126770b9a34, State: Initialized, Role: FOLLOWER
I20260806 23:55:44.140060 3061 consensus_queue.cc:260] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:44.141237 3061 ts_tablet_manager.cc:1428] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34: Time spent starting tablet: real 0.006s user 0.006s sys 0.003s
W20260806 23:55:44.190711 3217 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:55:44.114580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:59163)
0806 23:55:44.115098 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:44.115155 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:44.116282 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:44.189678 (+ 73396us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
W20260806 23:55:44.191264 3214 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:41925 (0 consecutive failures): Network error: Failed to ping master at 127.10.225.190:41925: Client connection negotiation failed: client connection to 127.10.225.190:41925: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 23:55:44.192060 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:44.191770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:33309)
0806 23:55:44.191912 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:44.191995 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 23:55:44.192800 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:44.192578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:58355)
0806 23:55:44.192691 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:44.192742 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 23:55:44.193030 3214 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 23:55:44.246491 3218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:44.246754 3218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:44.246814 3218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:44.251199 3218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:44.251271 3218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:44.251336 3218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:55:44.256789 3218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:38811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=39135
--tserver_master_addrs=127.10.225.190:41925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:44.257854 3218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:44.258870 3218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:44.269268 3236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:44.269280 3230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:44.269275 3233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:44.269316 3218 server_base.cc:1034] running on GCE node
I20260806 23:55:44.269811 3218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:44.270464 3218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:44.271638 3218 hybrid_clock.cc:648] HybridClock initialized: now 1786060544271609 us; error 38 us; skew 500 ppm
Aug 06 23:55:44 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:44.274461 3218 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:55:44.275382 3218 webserver.cc:460] Webserver started at http://127.10.225.130:39135/ using document root <none> and password file <none>
I20260806 23:55:44.275988 3218 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:44.276053 3218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:44.280033 3218 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 23:55:44.282055 3246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:44.283226 3218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:55:44.283362 3218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "8932d9a2a3d548a88298cbd1cd9f2476"
format_stamp: "Formatted at 2026-08-06 23:55:01 on dist-test-slave-nj21"
server_key: "208f4efd8498880f90ff784dc7377fa7"
server_key_iv: "d7c732eef9c3b98095d8b0700c822bca"
server_key_version: "encryptionkey@0"
I20260806 23:55:44.283936 3218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:44.302759 3218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:44.305919 3218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:44.306120 3218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:44.306751 3218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:55:44.307627 3218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:55:44.307708 3218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:44.307772 3218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:55:44.307797 3218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:44.317860 3218 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:38811
I20260806 23:55:44.317935 3433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:38811 every 8 connection(s)
I20260806 23:55:44.318912 3218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:55:44.322757 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:44.321043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:41599)
0806 23:55:44.321527 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:44.321633 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":133,"threads_started":1}
W20260806 23:55:44.323221 3435 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:41925 (0 consecutive failures): Network error: Failed to ping master at 127.10.225.190:41925: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
I20260806 23:55:44.323788 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:44.323569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:34971)
0806 23:55:44.323690 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:44.323735 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 23:55:44.324316 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:44.324158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:47835)
0806 23:55:44.324234 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:44.324272 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 23:55:44.324465 3435 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 23:55:44.326118 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 3218
I20260806 23:55:44.326535 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:33835
--local_ip_for_outbound_sockets=127.10.225.131
--tserver_master_addrs=127.10.225.190:41925
--webserver_port=39625
--webserver_interface=127.10.225.131
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:55:44.442639 3439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:55:44.442909 3439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:55:44.442951 3439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:55:44.446909 3439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:55:44.446971 3439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:55:44.447028 3439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:55:44.452194 3439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:33835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=39625
--tserver_master_addrs=127.10.225.190:41925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:55:44.453260 3439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:55:44.454272 3439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:55:44.463721 3454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:44.463721 3450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:55:44.463716 3452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:55:44.463760 3439 server_base.cc:1034] running on GCE node
I20260806 23:55:44.464335 3439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:55:44.464988 3439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:55:44.466145 3439 hybrid_clock.cc:648] HybridClock initialized: now 1786060544466139 us; error 34 us; skew 500 ppm
Aug 06 23:55:44 dist-test-slave-nj21 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:55:44.468852 3439 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:55:44.469885 3439 webserver.cc:460] Webserver started at http://127.10.225.131:39625/ using document root <none> and password file <none>
I20260806 23:55:44.470458 3439 fs_manager.cc:359] Metadata directory not provided
I20260806 23:55:44.470530 3439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:55:44.474455 3439 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260806 23:55:44.477028 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:44.478528 3439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 23:55:44.478648 3439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "322cfb9e5c5146cf8d296d858c4b6a9a"
format_stamp: "Formatted at 2026-08-06 23:55:02 on dist-test-slave-nj21"
server_key: "eddf4fc75d1798373fef18e4824d1c04"
server_key_iv: "2d7735e7142741d788b8d55ea93e5777"
server_key_version: "encryptionkey@0"
I20260806 23:55:44.479210 3439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:55:44.517108 3439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:55:44.520632 3439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:55:44.520838 3439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:55:44.521564 3439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:55:44.522445 3439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:55:44.522495 3439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:44.522552 3439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:55:44.522580 3439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:55:44.531780 3439 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:33835
I20260806 23:55:44.531800 3647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:33835 every 8 connection(s)
I20260806 23:55:44.532840 3439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:55:44.534891 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 3439
I20260806 23:55:44.536806 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:44.534940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:51093)
0806 23:55:44.535425 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:44.535531 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
W20260806 23:55:44.537245 3648 heartbeater.cc:644] Failed to heartbeat to 127.10.225.190:41925 (0 consecutive failures): Network error: Failed to ping master at 127.10.225.190:41925: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
I20260806 23:55:44.537856 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:44.537601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:56775)
0806 23:55:44.537748 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:44.537797 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 23:55:44.538401 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:44.538212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:43771)
0806 23:55:44.538312 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:44.538353 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 23:55:44.538576 3648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 23:55:45.194130 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:45.193655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:48129)
0806 23:55:45.193959 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:45.194027 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260806 23:55:45.325383 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:45.324975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:42267)
0806 23:55:45.325215 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:45.325269 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
I20260806 23:55:45.539966 3656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:45.539202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:56723)
0806 23:55:45.539780 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:45.539845 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":339,"threads_started":1}
I20260806 23:55:45.915380 3657 raft_consensus.cc:482] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:55:45.915648 3657 raft_consensus.cc:504] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:45.916831 3657 leader_election.cc:288] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811), 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835)
Aug 06 23:55:45 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.131@KRBTEST.COM
Aug 06 23:55:45 dist-test-slave-nj21 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.130@KRBTEST.COM
I20260806 23:55:45.930853 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:45.917413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:33835 (local address 127.10.225.129:48107)
0806 23:55:45.917797 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:45.917810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:45.917892 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:45.919355 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:45.919358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:45.919718 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:55:45.920251 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:45.920269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:45.921520 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:45.921523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:45.922479 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:45.922490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:45.923173 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:45.923176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:45.923384 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:45.924048 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:55:45.924064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:55:45.925712 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:55:45.928552 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:45.928554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:45.928556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:45.928783 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:45.929152 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:45.929156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:45.929158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:45.929262 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:45.929897 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:55:45.929904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:55:45.930124 (+ 220us) client_negotiation.cc:769] Sending connection context
0806 23:55:45.930290 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 23:55:45.930648 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":205,"spinlock_wait_cycles":20224,"thread_start_us":78,"threads_started":1}
I20260806 23:55:45.930907 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:45.917825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:48107 (local address 127.10.225.131:33835)
0806 23:55:45.918116 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:45.918136 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:45.918166 (+ 30us) server_negotiation.cc:407] Connection header received
0806 23:55:45.918765 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:45.918769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:45.919100 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:45.919383 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:45.920419 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:45.921377 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:45.922594 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:45.923049 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:45.923208 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:45.925946 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:45.925951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:45.925963 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:45.925992 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:45.927945 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:45.928901 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:45.928907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:45.928913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:45.929042 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:45.929357 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:45.929359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:45.929360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:45.929776 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:45.929900 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:45.930430 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 23:55:45.930747 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260806 23:55:45.930927 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:45.917021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:38811 (local address 127.10.225.129:55269)
0806 23:55:45.917277 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:45.917292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:55:45.917481 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:55:45.919165 (+ 1684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:55:45.919171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:55:45.919720 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:55:45.920251 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:45.920267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:45.921517 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:45.921519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:45.922482 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:55:45.922493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:55:45.923169 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:55:45.923172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:55:45.923381 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:45.924081 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:55:45.924093 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:55:45.926053 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:55:45.928106 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:45.928112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:45.928123 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:45.928443 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:45.928883 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:55:45.928886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:55:45.928888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:55:45.929051 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:55:45.929933 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:55:45.929936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:55:45.930125 (+ 189us) client_negotiation.cc:769] Sending connection context
0806 23:55:45.930291 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 23:55:45.930741 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":212,"spinlock_wait_cycles":37120,"thread_start_us":75,"threads_started":1}
I20260806 23:55:45.931456 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:45.917224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:55269 (local address 127.10.225.130:38811)
0806 23:55:45.917616 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0806 23:55:45.917630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:55:45.917665 (+ 35us) server_negotiation.cc:407] Connection header received
0806 23:55:45.918439 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:55:45.918447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:55:45.918793 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:55:45.919045 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:55:45.920426 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:45.921378 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:45.922606 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:55:45.923047 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:55:45.923192 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:55:45.926253 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:55:45.926260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:55:45.926273 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:55:45.926309 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:55:45.927959 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:45.928596 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:45.928602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:45.928606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:45.928785 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:55:45.929392 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:55:45.929396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:55:45.929398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:55:45.929822 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:55:45.929942 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 23:55:45.930427 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 23:55:45.930746 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
W20260806 23:55:45.933540 3057 leader_election.cc:341] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835): Not found: Tablet not found: c20ffabc0fd947f98704cc9290614e71
W20260806 23:55:45.933744 3057 leader_election.cc:341] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811): Not found: Tablet not found: c20ffabc0fd947f98704cc9290614e71
I20260806 23:55:45.933802 3057 leader_election.cc:302] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80c3d4caae9b4a3ea0a0f126770b9a34; no voters: 322cfb9e5c5146cf8d296d858c4b6a9a, 8932d9a2a3d548a88298cbd1cd9f2476
I20260806 23:55:45.934073 3657 raft_consensus.cc:2726] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:55:46.195246 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:46.194947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:33731)
0806 23:55:46.195100 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:46.195156 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 23:55:46.326917 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:46.326178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:44801)
0806 23:55:46.326612 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:46.326787 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":84,"threads_started":1}
I20260806 23:55:46.541124 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:46.540704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:60555)
0806 23:55:46.540968 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:46.541026 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260806 23:55:47.196786 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:47.196316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:39765)
0806 23:55:47.196615 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:47.196678 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":106,"threads_started":1}
I20260806 23:55:47.328382 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:47.327840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:49219)
0806 23:55:47.328177 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:47.328244 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":129,"threads_started":1}
I20260806 23:55:47.542342 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:47.541883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:33181)
0806 23:55:47.542179 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:47.542240 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":87,"threads_started":1}
I20260806 23:55:47.825592 3674 raft_consensus.cc:482] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:55:47.825728 3674 raft_consensus.cc:504] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:47.826144 3674 leader_election.cc:288] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811), 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835)
W20260806 23:55:47.827020 3057 leader_election.cc:341] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811): Not found: Tablet not found: c20ffabc0fd947f98704cc9290614e71
W20260806 23:55:47.827169 3057 leader_election.cc:341] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835): Not found: Tablet not found: c20ffabc0fd947f98704cc9290614e71
I20260806 23:55:47.827222 3057 leader_election.cc:302] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80c3d4caae9b4a3ea0a0f126770b9a34; no voters: 322cfb9e5c5146cf8d296d858c4b6a9a, 8932d9a2a3d548a88298cbd1cd9f2476
I20260806 23:55:47.827395 3674 raft_consensus.cc:2726] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:55:48.198138 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:48.197639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:41911)
0806 23:55:48.197939 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:48.198007 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":121,"threads_started":1}
I20260806 23:55:48.329737 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:48.329212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:57035)
0806 23:55:48.329548 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:48.329611 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
I20260806 23:55:48.543615 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:48.543166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:56021)
0806 23:55:48.543435 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:48.543506 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260806 23:55:49.199450 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:49.199003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:43587)
0806 23:55:49.199292 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:49.199349 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260806 23:55:49.330968 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:49.330536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:38061)
0806 23:55:49.330797 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:49.330853 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260806 23:55:49.545167 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:49.544478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:43165)
0806 23:55:49.544883 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:49.545054 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":82,"threads_started":1}
I20260806 23:55:50.200812 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:50.200354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:51875)
0806 23:55:50.200650 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:50.200710 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
I20260806 23:55:50.332307 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:50.331780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:42001)
0806 23:55:50.332078 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:50.332148 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260806 23:55:50.546494 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:50.546003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:41139)
0806 23:55:50.546323 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:50.546388 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260806 23:55:50.958463 3688 raft_consensus.cc:482] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:55:50.958611 3688 raft_consensus.cc:504] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:50.959055 3688 leader_election.cc:288] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811), 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835)
W20260806 23:55:50.959844 3057 leader_election.cc:341] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811): Not found: Tablet not found: c20ffabc0fd947f98704cc9290614e71
W20260806 23:55:50.959969 3057 leader_election.cc:341] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835): Not found: Tablet not found: c20ffabc0fd947f98704cc9290614e71
I20260806 23:55:50.960007 3057 leader_election.cc:302] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80c3d4caae9b4a3ea0a0f126770b9a34; no voters: 322cfb9e5c5146cf8d296d858c4b6a9a, 8932d9a2a3d548a88298cbd1cd9f2476
I20260806 23:55:50.960143 3688 raft_consensus.cc:2726] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:55:51.202317 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:51.201829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:52369)
0806 23:55:51.202141 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:51.202211 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
I20260806 23:55:51.333920 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:51.333104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:42841)
0806 23:55:51.333674 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:51.333778 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":77,"threads_started":1}
I20260806 23:55:51.547996 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:51.547430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:40557)
0806 23:55:51.547813 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:51.547888 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":82,"threads_started":1}
I20260806 23:55:52.204062 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:52.203496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:57007)
0806 23:55:52.203870 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:52.203944 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20260806 23:55:52.335369 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:52.334872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:42365)
0806 23:55:52.335178 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:52.335247 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260806 23:55:52.549209 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:52.548793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:45633)
0806 23:55:52.549050 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:52.549106 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260806 23:55:53.205475 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:53.204990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:36479)
0806 23:55:53.205306 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:53.205372 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
I20260806 23:55:53.336712 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:53.336243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:52017)
0806 23:55:53.336548 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:53.336609 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260806 23:55:53.550613 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:53.550115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:58837)
0806 23:55:53.550420 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:53.550493 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":84,"threads_started":1}
I20260806 23:55:54.206854 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:54.206416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:42495)
0806 23:55:54.206692 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:54.206753 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260806 23:55:54.338148 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:54.337669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:35977)
0806 23:55:54.337977 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:54.338046 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260806 23:55:54.551977 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:54.551469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:47365)
0806 23:55:54.551791 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:54.551871 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":78,"threads_started":1}
I20260806 23:55:55.208263 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:55.207773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:45253)
0806 23:55:55.208086 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:55.208156 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260806 23:55:55.339838 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:55.339203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:37021)
0806 23:55:55.339602 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:55.339706 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":165,"threads_started":1}
I20260806 23:55:55.539011 3719 raft_consensus.cc:482] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:55:55.539153 3719 raft_consensus.cc:504] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8932d9a2a3d548a88298cbd1cd9f2476" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 38811 } } peers { permanent_uuid: "322cfb9e5c5146cf8d296d858c4b6a9a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33835 } } peers { permanent_uuid: "80c3d4caae9b4a3ea0a0f126770b9a34" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 39259 } }
I20260806 23:55:55.539695 3719 leader_election.cc:288] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811), 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835)
W20260806 23:55:55.540591 3057 leader_election.cc:341] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8932d9a2a3d548a88298cbd1cd9f2476 (127.10.225.130:38811): Not found: Tablet not found: c20ffabc0fd947f98704cc9290614e71
W20260806 23:55:55.540724 3057 leader_election.cc:341] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 322cfb9e5c5146cf8d296d858c4b6a9a (127.10.225.131:33835): Not found: Tablet not found: c20ffabc0fd947f98704cc9290614e71
I20260806 23:55:55.540774 3057 leader_election.cc:302] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80c3d4caae9b4a3ea0a0f126770b9a34; no voters: 322cfb9e5c5146cf8d296d858c4b6a9a, 8932d9a2a3d548a88298cbd1cd9f2476
I20260806 23:55:55.540948 3719 raft_consensus.cc:2726] T c20ffabc0fd947f98704cc9290614e71 P 80c3d4caae9b4a3ea0a0f126770b9a34 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 23:55:55.553293 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:55.552808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:58609)
0806 23:55:55.553109 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:55.553178 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260806 23:55:56.209476 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:56.208997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:43277)
0806 23:55:56.209301 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:56.209365 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260806 23:55:56.341310 3722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:56.340762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:46741)
0806 23:55:56.341130 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:56.341201 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":165,"threads_started":1}
I20260806 23:55:56.554625 3723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:56.554152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:57085)
0806 23:55:56.554442 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:56.554501 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260806 23:55:57.210652 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:57.210235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:53503)
0806 23:55:57.210496 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:57.210552 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260806 23:55:57.342484 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:57.342077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:59001)
0806 23:55:57.342326 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:57.342378 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260806 23:55:57.555981 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:57.555477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:47167)
0806 23:55:57.555805 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:57.555868 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
I20260806 23:55:58.212028 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:58.211543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:45499)
0806 23:55:58.211856 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:58.211923 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":130,"threads_started":1}
I20260806 23:55:58.343868 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:58.343349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:36941)
0806 23:55:58.343663 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:58.343761 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":104,"threads_started":1}
I20260806 23:55:58.557227 3732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:58.556798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:57197)
0806 23:55:58.557069 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:58.557125 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260806 23:55:59.213486 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:59.213051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.129:41239)
0806 23:55:59.213333 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:59.213388 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
I20260806 23:55:59.345122 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:59.344700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.130:50533)
0806 23:55:59.344962 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:59.345023 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260806 23:55:59.539564 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 2966
I20260806 23:55:59.555334 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 3218
I20260806 23:55:59.562170 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:55:59.561764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41925 (local address 127.10.225.131:59821)
0806 23:55:59.562013 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 23:55:59.562069 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.190:41925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260806 23:55:59.566696 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 3439
2026-08-06T23:55:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59637 ms)
[----------] 30 tests from SecurityITest (259848 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:55:59 dist-test-slave-nj21 krb5kdc[3740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:55:59 dist-test-slave-nj21 krb5kdc[3740](info): set up 2 sockets
Aug 06 23:55:59 dist-test-slave-nj21 krb5kdc[3740](info): commencing operation
krb5kdc: starting...
W20260806 23:56:00.716861 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:00 dist-test-slave-nj21 krb5kdc[3740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:00.847361 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:33843
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:33843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:00.966264 3761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:00.966542 3761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:00.966584 3761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:00.970641 3761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:00.970709 3761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:00.970728 3761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:00.970743 3761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:00.970757 3761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:00.976219 3761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:33843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:33843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:00.977378 3761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:00.978413 3761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:00.987155 3771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:00.987133 3776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:00.987131 3769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:00.987118 3761 server_base.cc:1034] running on GCE node
I20260806 23:56:00.988037 3761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:00.989241 3761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:00.990423 3761 hybrid_clock.cc:648] HybridClock initialized: now 1786060560990404 us; error 39 us; skew 500 ppm
Aug 06 23:56:00 dist-test-slave-nj21 krb5kdc[3740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060560, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:00.993201 3761 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:00.994401 3761 webserver.cc:460] Webserver started at http://127.10.225.190:44445/ using document root <none> and password file <none>
I20260806 23:56:00.994957 3761 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:00.995016 3761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:00.995184 3761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:00.997316 3761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "f75120443c2e41958e158ad287ee8447"
format_stamp: "Formatted at 2026-08-06 23:56:00 on dist-test-slave-nj21"
server_key: "b01daf9f15b83d7152d537e9f9f83f25"
server_key_iv: "7908df3740befa77e524877c40f76abe"
server_key_version: "encryptionkey@0"
I20260806 23:56:00.997846 3761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "f75120443c2e41958e158ad287ee8447"
format_stamp: "Formatted at 2026-08-06 23:56:00 on dist-test-slave-nj21"
server_key: "b01daf9f15b83d7152d537e9f9f83f25"
server_key_iv: "7908df3740befa77e524877c40f76abe"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.001585 3761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:01.004002 3787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.005429 3761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:56:01.005555 3761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "f75120443c2e41958e158ad287ee8447"
format_stamp: "Formatted at 2026-08-06 23:56:00 on dist-test-slave-nj21"
server_key: "b01daf9f15b83d7152d537e9f9f83f25"
server_key_iv: "7908df3740befa77e524877c40f76abe"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.005648 3761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:01.018679 3761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:01.021682 3761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:01.021852 3761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:01.029441 3867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:33843 every 8 connection(s)
I20260806 23:56:01.029546 3761 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:33843
I20260806 23:56:01.030942 3761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:01.034178 3868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:01.035907 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 3761
I20260806 23:56:01.036031 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:01.036557 11142 external_mini_cluster.cc:1428] Setting key 9a3785b53f92175b78ff1dc3d3d2150f
I20260806 23:56:01.041679 3868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447: Bootstrap starting.
I20260806 23:56:01.044378 3868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:01.045213 3868 log.cc:824] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:56:01 dist-test-slave-nj21 krb5kdc[3740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:01.047291 3868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447: No bootstrap required, opened a new log
I20260806 23:56:01.051079 3868 raft_consensus.cc:348] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75120443c2e41958e158ad287ee8447" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 33843 } }
I20260806 23:56:01.051318 3868 raft_consensus.cc:374] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:01.051353 3868 raft_consensus.cc:729] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75120443c2e41958e158ad287ee8447, State: Initialized, Role: FOLLOWER
I20260806 23:56:01.051431 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.038104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.10.225.190:33843)
0806 23:56:01.038567 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.038586 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.038631 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:56:01.039693 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.039720 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.040158 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.040533 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:01.041225 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.042625 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.043510 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.044222 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.044390 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.046901 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:01.046906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:01.046920 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:01.046948 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:01.048606 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.049026 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.049031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.049035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.049149 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.049423 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.049425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.049427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.049850 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:01.050048 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.050325 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.050623 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":128,"threads_started":1}
I20260806 23:56:01.051867 3868 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75120443c2e41958e158ad287ee8447" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 33843 } }
I20260806 23:56:01.052030 3868 raft_consensus.cc:388] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:01.052096 3868 raft_consensus.cc:482] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:01.052193 3868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:01.053158 3868 raft_consensus.cc:504] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75120443c2e41958e158ad287ee8447" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 33843 } }
I20260806 23:56:01.053495 3868 leader_election.cc:302] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75120443c2e41958e158ad287ee8447; no voters:
I20260806 23:56:01.053857 3868 leader_election.cc:288] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:01.053956 3879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:01.054174 3879 raft_consensus.cc:686] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [term 1 LEADER]: Becoming Leader. State: Replica: f75120443c2e41958e158ad287ee8447, State: Running, Role: LEADER
I20260806 23:56:01.054538 3879 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75120443c2e41958e158ad287ee8447" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 33843 } }
I20260806 23:56:01.055114 3868 sys_catalog.cc:564] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:01.056360 3882 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75120443c2e41958e158ad287ee8447. Latest consensus state: current_term: 1 leader_uuid: "f75120443c2e41958e158ad287ee8447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75120443c2e41958e158ad287ee8447" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 33843 } } }
I20260806 23:56:01.056339 3881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f75120443c2e41958e158ad287ee8447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75120443c2e41958e158ad287ee8447" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 33843 } } }
I20260806 23:56:01.056535 3882 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:01.056536 3881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:01.056949 3892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:01.059803 3892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:01.066125 3892 catalog_manager.cc:1347] Generated new cluster ID: 34e321a5f11c425e8b1e4e7b5a1f5409
I20260806 23:56:01.066205 3892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:01.081956 3892 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:01.083372 3892 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:01.091933 3892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447: Generated new TSK 0
I20260806 23:56:01.092866 3892 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:56:01.128134 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:33843
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:01.253041 3908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:01.253328 3908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:01.253376 3908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:01.257618 3908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:01.257689 3908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:01.257752 3908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:56:01.263244 3908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:33843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:01.264408 3908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:01.265448 3908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:01.275127 3924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:01.275312 3908 server_base.cc:1034] running on GCE node
W20260806 23:56:01.275203 3917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:01.275538 3918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:01.275962 3908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:01.276621 3908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:01.277793 3908 hybrid_clock.cc:648] HybridClock initialized: now 1786060561277777 us; error 33 us; skew 500 ppm
Aug 06 23:56:01 dist-test-slave-nj21 krb5kdc[3740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060561, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:01.280686 3908 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:56:01.281579 3908 webserver.cc:460] Webserver started at http://127.10.225.129:37659/ using document root <none> and password file <none>
I20260806 23:56:01.282141 3908 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:01.282208 3908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:01.282423 3908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:01.284550 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "e921f64ac973409295f3f6e44ce75109"
format_stamp: "Formatted at 2026-08-06 23:56:01 on dist-test-slave-nj21"
server_key: "d1cf0abc62a90e4e4ea90d378d7c8234"
server_key_iv: "c460aa5b96aea98d0f0485629f1f590a"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.285115 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "e921f64ac973409295f3f6e44ce75109"
format_stamp: "Formatted at 2026-08-06 23:56:01 on dist-test-slave-nj21"
server_key: "d1cf0abc62a90e4e4ea90d378d7c8234"
server_key_iv: "c460aa5b96aea98d0f0485629f1f590a"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.289188 3908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:56:01.291870 3936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.293104 3908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:01.293218 3908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "e921f64ac973409295f3f6e44ce75109"
format_stamp: "Formatted at 2026-08-06 23:56:01 on dist-test-slave-nj21"
server_key: "d1cf0abc62a90e4e4ea90d378d7c8234"
server_key_iv: "c460aa5b96aea98d0f0485629f1f590a"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.293325 3908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:01.304716 3908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:01.308302 3908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:01.308488 3908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:01.309075 3908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:01.310055 3908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:01.310102 3908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.310155 3908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:01.310184 3908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.318652 3908 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:45097
I20260806 23:56:01.318787 4113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:45097 every 8 connection(s)
I20260806 23:56:01.319619 3908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:56:01.328348 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 3908
I20260806 23:56:01.328508 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:01.328864 11142 external_mini_cluster.cc:1428] Setting key fbe52096488324646483271da756a81e
Aug 06 23:56:01 dist-test-slave-nj21 krb5kdc[3740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060561, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:01.334350 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.321544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:42519 (local address 127.10.225.190:33843)
0806 23:56:01.321697 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.321701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.322513 (+ 812us) server_negotiation.cc:407] Connection header received
0806 23:56:01.323860 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.323863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.323924 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.323999 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:01.325174 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.325704 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.326818 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.327047 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.327133 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.330356 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:01.330359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:01.330361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:01.330384 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:01.332042 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.332584 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.332588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.332590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.332691 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.333106 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.333108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.333110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.333322 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:01.333412 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.334090 (+ 678us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.334206 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 23:56:01.335402 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.321864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:33843 (local address 127.10.225.129:42519)
0806 23:56:01.322362 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:01.322405 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:01.323628 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:01.324178 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:01.324185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:01.324538 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:01.325032 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.325046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.325822 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.325824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.326695 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.326705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.327192 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.327196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.327364 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.328186 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:01.328207 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:01.330182 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:01.332174 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:01.332180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:01.332194 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:01.332482 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:01.332784 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:01.332787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:01.332789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:01.332990 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:01.333426 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:01.333433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:01.333777 (+ 344us) client_negotiation.cc:769] Sending connection context
0806 23:56:01.334011 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 23:56:01.334368 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260806 23:56:01.336772 4114 heartbeater.cc:347] Connected to a master server at 127.10.225.190:33843
I20260806 23:56:01.337025 4114 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:01.337539 4114 heartbeater.cc:511] Master 127.10.225.190:33843 requested a full tablet report, sending...
I20260806 23:56:01.339094 3811 ts_manager.cc:194] Registered new tserver with Master: e921f64ac973409295f3f6e44ce75109 (127.10.225.129:45097)
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:56:01.340332 3811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:42519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:56:01.356794 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:33843
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:01.472667 4122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:01.473031 4122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:01.473102 4122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:01.477939 4122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:01.478008 4122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:01.478070 4122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:56:01.483475 4122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:33843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:01.484575 4122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:01.485569 4122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:01.495133 4129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:01.495186 4122 server_base.cc:1034] running on GCE node
W20260806 23:56:01.495121 4134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:01.495093 4127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:01.495759 4122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:01.496490 4122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:01.497678 4122 hybrid_clock.cc:648] HybridClock initialized: now 1786060561497653 us; error 34 us; skew 500 ppm
Aug 06 23:56:01 dist-test-slave-nj21 krb5kdc[3740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060561, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:01.500438 4122 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:56:01.501287 4122 webserver.cc:460] Webserver started at http://127.10.225.130:34141/ using document root <none> and password file <none>
I20260806 23:56:01.501842 4122 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:01.501909 4122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:01.502122 4122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:01.504316 4122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "44ea119ca6bc4cfdabb3001b564f3572"
format_stamp: "Formatted at 2026-08-06 23:56:01 on dist-test-slave-nj21"
server_key: "c14df91007917687446f18d7e6d588ab"
server_key_iv: "4242990da1a031cc39b661f2a699105b"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.504873 4122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "44ea119ca6bc4cfdabb3001b564f3572"
format_stamp: "Formatted at 2026-08-06 23:56:01 on dist-test-slave-nj21"
server_key: "c14df91007917687446f18d7e6d588ab"
server_key_iv: "4242990da1a031cc39b661f2a699105b"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.508613 4122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:56:01.511286 4143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.512576 4122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:01.512688 4122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "44ea119ca6bc4cfdabb3001b564f3572"
format_stamp: "Formatted at 2026-08-06 23:56:01 on dist-test-slave-nj21"
server_key: "c14df91007917687446f18d7e6d588ab"
server_key_iv: "4242990da1a031cc39b661f2a699105b"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.512787 4122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:01.551810 4122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:01.554841 4122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:01.555022 4122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:01.555691 4122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:01.556663 4122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:01.556708 4122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.556751 4122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:01.556772 4122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.565661 4122 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:35927
I20260806 23:56:01.565686 4305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:35927 every 8 connection(s)
I20260806 23:56:01.566527 4122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:56:01.574952 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 4122
I20260806 23:56:01.575093 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:01.575529 11142 external_mini_cluster.cc:1428] Setting key eb67d33a2dbb5cad6e4532fdccffa281
Aug 06 23:56:01 dist-test-slave-nj21 krb5kdc[3740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060561, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:01.579969 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.568299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:54781 (local address 127.10.225.190:33843)
0806 23:56:01.568443 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.568448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.569274 (+ 826us) server_negotiation.cc:407] Connection header received
0806 23:56:01.570636 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.570639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.570688 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.570750 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:01.571929 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.572469 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.573576 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.573789 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.573861 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.576433 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:01.576437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:01.576439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:01.576465 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:01.577848 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.578422 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.578424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.578426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.578503 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.578839 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.578840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.578841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.579051 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:01.579157 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.579664 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.579810 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:56:01.580780 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.568684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:33843 (local address 127.10.225.130:54781)
0806 23:56:01.569130 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:01.569171 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:01.570388 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:01.570895 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:01.570901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:01.571234 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:01.571775 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.571792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.572599 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.572602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.573481 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.573491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.573902 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.573905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.574045 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.574557 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:01.574573 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:01.576259 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:01.577963 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:01.577969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:01.577985 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:01.578319 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:01.578584 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:01.578587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:01.578589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:01.578761 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:01.579164 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:01.579171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:01.579439 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 23:56:01.579599 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 23:56:01.579914 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260806 23:56:01.581866 4306 heartbeater.cc:347] Connected to a master server at 127.10.225.190:33843
I20260806 23:56:01.582098 4306 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:01.582587 4306 heartbeater.cc:511] Master 127.10.225.190:33843 requested a full tablet report, sending...
I20260806 23:56:01.583701 3811 ts_manager.cc:194] Registered new tserver with Master: 44ea119ca6bc4cfdabb3001b564f3572 (127.10.225.130:35927)
I20260806 23:56:01.584321 3811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:54781
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:56:01.604324 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:33843
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:01.724599 4311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:01.724882 4311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:01.724948 4311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:01.728986 4311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:01.729049 4311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:01.729135 4311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:56:01.734628 4311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:33843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:01.735760 4311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:01.736783 4311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:01.746500 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:01.746511 4327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:01.746599 4311 server_base.cc:1034] running on GCE node
W20260806 23:56:01.746519 4330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:01.747129 4311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:01.747774 4311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:01.748932 4311 hybrid_clock.cc:648] HybridClock initialized: now 1786060561748916 us; error 27 us; skew 500 ppm
Aug 06 23:56:01 dist-test-slave-nj21 krb5kdc[3740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060561, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:01.751631 4311 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:56:01.752605 4311 webserver.cc:460] Webserver started at http://127.10.225.131:39655/ using document root <none> and password file <none>
I20260806 23:56:01.753167 4311 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:01.753229 4311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:01.753444 4311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:01.755620 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "0f17c915632c48e4ac8ec21d7e87b3c0"
format_stamp: "Formatted at 2026-08-06 23:56:01 on dist-test-slave-nj21"
server_key: "863ca05e3c1c79c231458fa0f433468f"
server_key_iv: "a84bf58ac24579eb08289bc1e19c7187"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.756242 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "0f17c915632c48e4ac8ec21d7e87b3c0"
format_stamp: "Formatted at 2026-08-06 23:56:01 on dist-test-slave-nj21"
server_key: "863ca05e3c1c79c231458fa0f433468f"
server_key_iv: "a84bf58ac24579eb08289bc1e19c7187"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.759994 4311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:56:01.762413 4345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.763827 4311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:01.763942 4311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "0f17c915632c48e4ac8ec21d7e87b3c0"
format_stamp: "Formatted at 2026-08-06 23:56:01 on dist-test-slave-nj21"
server_key: "863ca05e3c1c79c231458fa0f433468f"
server_key_iv: "a84bf58ac24579eb08289bc1e19c7187"
server_key_version: "encryptionkey@0"
I20260806 23:56:01.764055 4311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:01.777637 4311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:01.780715 4311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:01.780900 4311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:01.781513 4311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:01.782377 4311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:01.782423 4311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.782469 4311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:01.782497 4311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:01.791335 4311 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:37781
I20260806 23:56:01.791428 4534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:37781 every 8 connection(s)
I20260806 23:56:01.792300 4311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:56:01.801971 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 4311
I20260806 23:56:01.802115 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
Aug 06 23:56:01 dist-test-slave-nj21 krb5kdc[3740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060561, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:01.802454 11142 external_mini_cluster.cc:1428] Setting key ac168a74163653e81b6fa58ade196ca5
I20260806 23:56:01.806406 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.794181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:39811 (local address 127.10.225.190:33843)
0806 23:56:01.794480 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.794485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.795359 (+ 874us) server_negotiation.cc:407] Connection header received
0806 23:56:01.796740 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.796744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.796805 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.796882 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:01.798240 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.798829 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.800023 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.800235 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.800322 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.802808 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:01.802811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:01.802813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:01.802838 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:01.804375 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.804970 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.804972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.804974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.805056 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.805419 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.805421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.805422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.805612 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:01.805689 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.806178 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.806275 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 23:56:01.807240 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.794612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:33843 (local address 127.10.225.131:39811)
0806 23:56:01.795125 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:01.795167 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:01.796528 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:01.797055 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:01.797061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:01.797572 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:01.798078 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.798092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.798960 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.798963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.799913 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.799924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.800372 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.800376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.800536 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.801070 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:01.801086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:01.802633 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:01.804504 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:01.804508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:01.804521 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:01.804876 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:01.805167 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:01.805170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:01.805171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:01.805334 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:01.805723 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:01.805729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:01.805984 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 23:56:01.806144 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 23:56:01.806415 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260806 23:56:01.808416 4535 heartbeater.cc:347] Connected to a master server at 127.10.225.190:33843
I20260806 23:56:01.808648 4535 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:01.809157 4535 heartbeater.cc:511] Master 127.10.225.190:33843 requested a full tablet report, sending...
I20260806 23:56:01.810223 3811 ts_manager.cc:194] Registered new tserver with Master: 0f17c915632c48e4ac8ec21d7e87b3c0 (127.10.225.131:37781)
I20260806 23:56:01.810827 3811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:39811
I20260806 23:56:01.816618 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:56:01.826045 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.818684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.10.225.190:33843)
0806 23:56:01.818819 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.818823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.819020 (+ 197us) server_negotiation.cc:407] Connection header received
0806 23:56:01.819083 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.819086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.819129 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.819202 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:01.819892 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.820417 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.821357 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.821578 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.821671 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.823088 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:01.823092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:01.823094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:01.823123 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:01.824439 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.824821 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.824824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.824825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.824903 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:01.825144 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:01.825147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:01.825148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:01.825434 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:01.825544 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.825783 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.825888 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:56:01.829039 3811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:56:01.832854 3811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:01.847774 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.842388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.10.225.130:35927)
0806 23:56:01.842781 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.842793 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.842981 (+ 188us) server_negotiation.cc:407] Connection header received
0806 23:56:01.843209 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.843215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.843396 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.843530 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:01.843784 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.844769 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.846046 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.846905 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.847041 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.847177 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.847477 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.847580 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":96,"threads_started":1}
I20260806 23:56:01.847803 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.841421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45097 (local address 127.0.0.1:36484)
0806 23:56:01.841902 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:01.841950 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:01.842071 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:01.842809 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:01.842815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:01.842837 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:01.843002 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.843011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.844606 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.844609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.846019 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.846031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.847059 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.847063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.847237 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.847313 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 23:56:01.847432 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 23:56:01.847582 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":78,"thread_start_us":100,"threads_started":1}
I20260806 23:56:01.847803 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.842044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35927 (local address 127.0.0.1:55740)
0806 23:56:01.842897 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:01.842909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:01.843051 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:01.843547 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:01.843550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:01.843567 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:01.843663 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.843697 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.844893 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.844896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.845933 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.845944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.847059 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.847063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.847239 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.847311 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 23:56:01.847432 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 23:56:01.847582 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":78,"thread_start_us":102,"threads_started":1}
I20260806 23:56:01.848242 4555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.841577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.10.225.129:45097)
0806 23:56:01.842317 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.842324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.842348 (+ 24us) server_negotiation.cc:407] Connection header received
0806 23:56:01.842427 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.842433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.842645 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.843156 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:01.843288 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.844461 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.846151 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.846933 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.847041 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.847177 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.847996 (+ 819us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.848097 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":139,"threads_started":1}
I20260806 23:56:01.848554 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.841566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:37781 (local address 127.0.0.1:47318)
0806 23:56:01.842608 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:01.842623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:01.842712 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:01.843371 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:01.843374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:01.843391 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:01.843495 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.843502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.844531 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.844534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.846021 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.846031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.848194 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.848197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.848287 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.848313 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 23:56:01.848363 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 23:56:01.848432 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":140,"thread_start_us":94,"threads_started":1}
I20260806 23:56:01.849255 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.842241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.10.225.131:37781)
0806 23:56:01.842981 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.842995 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.843012 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:56:01.843087 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.843092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.843255 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.843384 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:01.843618 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.844403 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.847242 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.848058 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.848835 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.848981 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.849029 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.849119 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":141,"threads_started":1}
I20260806 23:56:01.851377 4212 tablet_service.cc:1467] Processing CreateTablet for tablet 06a6f2a38fd048b7a019f0f4c61464da (DEFAULT_TABLE table=test-table [id=d3ab87221ee648ca928ec44bddb39616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:01.851444 4017 tablet_service.cc:1467] Processing CreateTablet for tablet 06a6f2a38fd048b7a019f0f4c61464da (DEFAULT_TABLE table=test-table [id=d3ab87221ee648ca928ec44bddb39616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:01.851714 4418 tablet_service.cc:1467] Processing CreateTablet for tablet 06a6f2a38fd048b7a019f0f4c61464da (DEFAULT_TABLE table=test-table [id=d3ab87221ee648ca928ec44bddb39616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:01.852535 4212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a6f2a38fd048b7a019f0f4c61464da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:01.852541 4017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a6f2a38fd048b7a019f0f4c61464da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:01.852572 4418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a6f2a38fd048b7a019f0f4c61464da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:01.859232 4559 tablet_bootstrap.cc:492] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: Bootstrap starting.
I20260806 23:56:01.860597 4560 tablet_bootstrap.cc:492] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109: Bootstrap starting.
I20260806 23:56:01.861153 4561 tablet_bootstrap.cc:492] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: Bootstrap starting.
I20260806 23:56:01.861769 4559 tablet_bootstrap.cc:654] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:01.862716 4560 tablet_bootstrap.cc:654] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:01.862824 4561 tablet_bootstrap.cc:654] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:01.863180 4559 log.cc:824] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:01.863539 4560 log.cc:824] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:01.864535 4561 log.cc:824] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:01.865684 4560 tablet_bootstrap.cc:492] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109: No bootstrap required, opened a new log
I20260806 23:56:01.865736 4559 tablet_bootstrap.cc:492] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: No bootstrap required, opened a new log
I20260806 23:56:01.865886 4560 ts_tablet_manager.cc:1397] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 23:56:01.865940 4559 ts_tablet_manager.cc:1397] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 23:56:01.866657 4561 tablet_bootstrap.cc:492] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: No bootstrap required, opened a new log
I20260806 23:56:01.866827 4561 ts_tablet_manager.cc:1397] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:56:01.869505 4560 raft_consensus.cc:348] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } }
I20260806 23:56:01.869743 4560 raft_consensus.cc:374] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:01.869827 4560 raft_consensus.cc:729] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e921f64ac973409295f3f6e44ce75109, State: Initialized, Role: FOLLOWER
I20260806 23:56:01.869921 4559 raft_consensus.cc:348] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } }
I20260806 23:56:01.870172 4559 raft_consensus.cc:374] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:01.870265 4559 raft_consensus.cc:729] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ea119ca6bc4cfdabb3001b564f3572, State: Initialized, Role: FOLLOWER
I20260806 23:56:01.870554 4561 raft_consensus.cc:348] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } }
I20260806 23:56:01.870652 4560 consensus_queue.cc:260] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } }
I20260806 23:56:01.870752 4559 consensus_queue.cc:260] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } }
I20260806 23:56:01.870932 4561 raft_consensus.cc:374] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:01.871029 4561 raft_consensus.cc:729] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f17c915632c48e4ac8ec21d7e87b3c0, State: Initialized, Role: FOLLOWER
I20260806 23:56:01.871457 4561 consensus_queue.cc:260] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } }
I20260806 23:56:01.871742 4560 ts_tablet_manager.cc:1428] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260806 23:56:01.871771 4559 ts_tablet_manager.cc:1428] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:56:01.872097 4114 heartbeater.cc:503] Master 127.10.225.190:33843 was elected leader, sending a full tablet report...
I20260806 23:56:01.872152 4306 heartbeater.cc:503] Master 127.10.225.190:33843 was elected leader, sending a full tablet report...
I20260806 23:56:01.873438 4561 ts_tablet_manager.cc:1428] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 23:56:01.873740 4535 heartbeater.cc:503] Master 127.10.225.190:33843 was elected leader, sending a full tablet report...
I20260806 23:56:01.895861 4569 raft_consensus.cc:482] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:01.896114 4569 raft_consensus.cc:504] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } }
I20260806 23:56:01.897614 4569 leader_election.cc:288] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e921f64ac973409295f3f6e44ce75109 (127.10.225.129:45097), 44ea119ca6bc4cfdabb3001b564f3572 (127.10.225.130:35927)
I20260806 23:56:01.901368 4555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.897969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:60879 (local address 127.10.225.129:45097)
0806 23:56:01.898090 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.898095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.898196 (+ 101us) server_negotiation.cc:407] Connection header received
0806 23:56:01.898358 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.898361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.898406 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.898469 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:01.898794 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.899300 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.900421 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.900840 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.900957 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.901015 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.901180 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.901252 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 23:56:01.901367 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.897872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45097 (local address 127.10.225.131:60879)
0806 23:56:01.898096 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:01.898111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:01.898224 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:01.898533 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:01.898536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:01.898555 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:01.898684 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.898691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.899419 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.899422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.900300 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.900310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.900957 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.900961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.901068 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.901090 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:56:01.901155 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:56:01.901227 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260806 23:56:01.902064 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.898001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35927 (local address 127.10.225.131:47211)
0806 23:56:01.898384 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:01.898395 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:01.898486 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:01.898721 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:01.898723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:01.898738 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:01.898853 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.898858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.899883 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.899888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.900863 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:01.900874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.901616 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.901620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:01.901753 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.901779 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 23:56:01.901852 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 23:56:01.901935 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":132,"threads_started":1}
I20260806 23:56:01.902083 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.898072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:47211 (local address 127.10.225.130:35927)
0806 23:56:01.898209 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.898213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.898472 (+ 259us) server_negotiation.cc:407] Connection header received
0806 23:56:01.898599 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.898601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.898644 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.898704 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:01.898969 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.899628 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.901015 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.901434 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.901554 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.901599 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.901881 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.901961 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:56:01.902079 4045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a6f2a38fd048b7a019f0f4c61464da" candidate_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e921f64ac973409295f3f6e44ce75109" is_pre_election: true
I20260806 23:56:01.902433 4045 raft_consensus.cc:2454] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f17c915632c48e4ac8ec21d7e87b3c0 in term 0.
I20260806 23:56:01.902958 4241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a6f2a38fd048b7a019f0f4c61464da" candidate_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ea119ca6bc4cfdabb3001b564f3572" is_pre_election: true
I20260806 23:56:01.903003 4347 leader_election.cc:302] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f17c915632c48e4ac8ec21d7e87b3c0, e921f64ac973409295f3f6e44ce75109; no voters:
I20260806 23:56:01.903208 4241 raft_consensus.cc:2454] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f17c915632c48e4ac8ec21d7e87b3c0 in term 0.
I20260806 23:56:01.903272 4569 raft_consensus.cc:2781] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:01.903369 4569 raft_consensus.cc:482] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:01.903417 4569 raft_consensus.cc:3037] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:01.904511 4569 raft_consensus.cc:504] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } }
I20260806 23:56:01.904837 4569 leader_election.cc:288] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [CANDIDATE]: Term 1 election: Requested vote from peers e921f64ac973409295f3f6e44ce75109 (127.10.225.129:45097), 44ea119ca6bc4cfdabb3001b564f3572 (127.10.225.130:35927)
I20260806 23:56:01.905193 4045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a6f2a38fd048b7a019f0f4c61464da" candidate_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e921f64ac973409295f3f6e44ce75109"
I20260806 23:56:01.905193 4241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a6f2a38fd048b7a019f0f4c61464da" candidate_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ea119ca6bc4cfdabb3001b564f3572"
I20260806 23:56:01.905320 4045 raft_consensus.cc:3037] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:01.905320 4241 raft_consensus.cc:3037] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:01.906738 4045 raft_consensus.cc:2454] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f17c915632c48e4ac8ec21d7e87b3c0 in term 1.
I20260806 23:56:01.906747 4241 raft_consensus.cc:2454] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f17c915632c48e4ac8ec21d7e87b3c0 in term 1.
I20260806 23:56:01.907050 4346 leader_election.cc:302] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f17c915632c48e4ac8ec21d7e87b3c0, 44ea119ca6bc4cfdabb3001b564f3572; no voters:
I20260806 23:56:01.907220 4569 raft_consensus.cc:2781] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:01.907486 4569 raft_consensus.cc:686] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 1 LEADER]: Becoming Leader. State: Replica: 0f17c915632c48e4ac8ec21d7e87b3c0, State: Running, Role: LEADER
I20260806 23:56:01.907840 4569 consensus_queue.cc:237] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } }
I20260806 23:56:01.911224 3808 catalog_manager.cc:5495] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f17c915632c48e4ac8ec21d7e87b3c0 (127.10.225.131). New cstate: current_term: 1 leader_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f17c915632c48e4ac8ec21d7e87b3c0" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 37781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 } health_report { overall_health: UNKNOWN } } }
I20260806 23:56:01.926844 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.923202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.10.225.131:37781)
0806 23:56:01.923339 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:01.923343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:01.923357 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:56:01.923469 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:01.923471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:01.923529 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:01.923613 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:01.923888 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.924394 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.925289 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:01.925527 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:01.925614 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:01.925953 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:01.926393 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:01.926498 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:01.926618 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 23:56:01.926676 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:56:01.932750 4045 raft_consensus.cc:1261] T 06a6f2a38fd048b7a019f0f4c61464da P e921f64ac973409295f3f6e44ce75109 [term 1 FOLLOWER]: Refusing update from remote peer 0f17c915632c48e4ac8ec21d7e87b3c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:01.932750 4241 raft_consensus.cc:1261] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [term 1 FOLLOWER]: Refusing update from remote peer 0f17c915632c48e4ac8ec21d7e87b3c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:01.933480 4571 consensus_queue.cc:1035] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e921f64ac973409295f3f6e44ce75109" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:01.933656 4569 consensus_queue.cc:1035] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ea119ca6bc4cfdabb3001b564f3572" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:01.942278 4578 mvcc.cc:205] Tried to move back new op lower bound from 7315704061669539840 to 7315704061577326592. Current Snapshot: MvccSnapshot[applied={T|T < 7315704061669539840}]
I20260806 23:56:01.950434 3808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:01.950635 3808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:01.953179 3808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f75120443c2e41958e158ad287ee8447: Sending DeleteTablet for 3 replicas of tablet 06a6f2a38fd048b7a019f0f4c61464da
I20260806 23:56:01.954008 4418 tablet_service.cc:1514] Processing DeleteTablet for tablet 06a6f2a38fd048b7a019f0f4c61464da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:01 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:47318
I20260806 23:56:01.954015 4017 tablet_service.cc:1514] Processing DeleteTablet for tablet 06a6f2a38fd048b7a019f0f4c61464da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:01 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:36484
I20260806 23:56:01.954522 4212 tablet_service.cc:1514] Processing DeleteTablet for tablet 06a6f2a38fd048b7a019f0f4c61464da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:01 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:55740
I20260806 23:56:01.954828 4595 tablet_replica.cc:330] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: stopping tablet replica
I20260806 23:56:01.954905 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 3908
I20260806 23:56:01.955881 4595 raft_consensus.cc:2229] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:56:01.956342 4595 raft_consensus.cc:2258] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:01.959390 4595 ts_tablet_manager.cc:1905] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:01.961974 4600 tablet_replica.cc:330] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: stopping tablet replica
I20260806 23:56:01.962390 4600 raft_consensus.cc:2229] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:01.962459 4595 ts_tablet_manager.cc:1918] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:01.962620 4595 log.cc:1196] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/06a6f2a38fd048b7a019f0f4c61464da
I20260806 23:56:01.962809 4600 raft_consensus.cc:2258] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:01.963022 4595 ts_tablet_manager.cc:1939] T 06a6f2a38fd048b7a019f0f4c61464da P 0f17c915632c48e4ac8ec21d7e87b3c0: Deleting consensus metadata
I20260806 23:56:01.964373 3793 catalog_manager.cc:4841] TS 0f17c915632c48e4ac8ec21d7e87b3c0 (127.10.225.131:37781): tablet 06a6f2a38fd048b7a019f0f4c61464da (table test-table [id=d3ab87221ee648ca928ec44bddb39616]) successfully deleted
I20260806 23:56:01.964524 4600 ts_tablet_manager.cc:1905] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:01.967104 4600 ts_tablet_manager.cc:1918] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:01.967199 4600 log.cc:1196] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/06a6f2a38fd048b7a019f0f4c61464da
I20260806 23:56:01.967522 4600 ts_tablet_manager.cc:1939] T 06a6f2a38fd048b7a019f0f4c61464da P 44ea119ca6bc4cfdabb3001b564f3572: Deleting consensus metadata
I20260806 23:56:01.968994 3789 catalog_manager.cc:4841] TS 44ea119ca6bc4cfdabb3001b564f3572 (127.10.225.130:35927): tablet 06a6f2a38fd048b7a019f0f4c61464da (table test-table [id=d3ab87221ee648ca928ec44bddb39616]) successfully deleted
I20260806 23:56:01.969070 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 4122
W20260806 23:56:01.969012 3790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:45097 (error 108)
I20260806 23:56:01.976045 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:01.975596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45097 (local address 127.0.0.1:36490)
0806 23:56:01.975851 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:01.975931 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260806 23:56:01.976506 3790 catalog_manager.cc:4584] TS e921f64ac973409295f3f6e44ce75109 (127.10.225.129:45097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06a6f2a38fd048b7a019f0f4c61464da: Network error: Client connection negotiation failed: client connection to 127.10.225.129:45097: connect: Connection refused (error 111)
I20260806 23:56:01.985635 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 4311
I20260806 23:56:01.999852 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 3761
2026-08-06T23:56:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2426 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T23:56:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:02Z Disabled control of system clock
I20260806 23:56:02.034689 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:37403
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:37403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 23:56:02.147967 4607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:02.148252 4607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:02.148290 4607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:02.152381 4607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:02.152441 4607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:02.152455 4607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:02.152470 4607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:02.152486 4607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:02.157302 4607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:37403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:37403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:02.158479 4607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:02.159943 4607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:02.168411 4619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:02.168565 4620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:02.168565 4622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:02.168735 4607 server_base.cc:1034] running on GCE node
I20260806 23:56:02.169731 4607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:02.170881 4607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:02.172039 4607 hybrid_clock.cc:648] HybridClock initialized: now 1786060562172023 us; error 28 us; skew 500 ppm
I20260806 23:56:02.173919 4607 webserver.cc:460] Webserver started at http://127.10.225.190:41843/ using document root <none> and password file <none>
I20260806 23:56:02.174491 4607 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:02.174561 4607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:02.174779 4607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:02.176937 4607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "afee4357bf7845a28ba98ab2842d7ec1"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "7afa09763756b1a5649d58b9c5e44030"
server_key_iv: "9c52c27da5785a0d03b077577a1612ba"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.177505 4607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "afee4357bf7845a28ba98ab2842d7ec1"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "7afa09763756b1a5649d58b9c5e44030"
server_key_iv: "9c52c27da5785a0d03b077577a1612ba"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.181632 4607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:02.184156 4631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.185398 4607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:02.185523 4607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "afee4357bf7845a28ba98ab2842d7ec1"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "7afa09763756b1a5649d58b9c5e44030"
server_key_iv: "9c52c27da5785a0d03b077577a1612ba"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.185621 4607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:02.199245 4607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:02.200044 4607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:02.200217 4607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:02.207058 4607 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:37403
I20260806 23:56:02.207083 4721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:37403 every 8 connection(s)
I20260806 23:56:02.208036 4607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:02.210975 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 4607
I20260806 23:56:02.211115 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:02.211380 4724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:02.211448 11142 external_mini_cluster.cc:1428] Setting key 50d0235c1d7c9b8f4eb77293efce6a1a
I20260806 23:56:02.217727 4724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1: Bootstrap starting.
I20260806 23:56:02.219967 4724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:02.220686 4724 log.cc:824] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:02.220709 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.212889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.10.225.190:37403)
0806 23:56:02.213547 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:02.213561 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:02.213595 (+ 34us) server_negotiation.cc:407] Connection header received
0806 23:56:02.214226 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:02.214242 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:02.214621 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:02.214915 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:02.215351 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.216353 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.217230 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.217696 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.217845 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.218970 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:02.218974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:02.218987 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:02.219374 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:02.219430 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:02.219451 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:02.219545 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:02.219709 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 23:56:02.219966 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":152,"threads_started":1}
I20260806 23:56:02.222651 4724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1: No bootstrap required, opened a new log
I20260806 23:56:02.226246 4724 raft_consensus.cc:348] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee4357bf7845a28ba98ab2842d7ec1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37403 } }
I20260806 23:56:02.226466 4724 raft_consensus.cc:374] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:02.226516 4724 raft_consensus.cc:729] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afee4357bf7845a28ba98ab2842d7ec1, State: Initialized, Role: FOLLOWER
I20260806 23:56:02.226926 4724 consensus_queue.cc:260] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee4357bf7845a28ba98ab2842d7ec1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37403 } }
I20260806 23:56:02.227052 4724 raft_consensus.cc:388] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:02.227114 4724 raft_consensus.cc:482] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:02.227201 4724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:02.228144 4724 raft_consensus.cc:504] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee4357bf7845a28ba98ab2842d7ec1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37403 } }
I20260806 23:56:02.228462 4724 leader_election.cc:302] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afee4357bf7845a28ba98ab2842d7ec1; no voters:
I20260806 23:56:02.228750 4724 leader_election.cc:288] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:02.228844 4729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:02.229029 4729 raft_consensus.cc:686] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [term 1 LEADER]: Becoming Leader. State: Replica: afee4357bf7845a28ba98ab2842d7ec1, State: Running, Role: LEADER
I20260806 23:56:02.229413 4729 consensus_queue.cc:237] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee4357bf7845a28ba98ab2842d7ec1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37403 } }
I20260806 23:56:02.230166 4724 sys_catalog.cc:564] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:02.231153 4730 sys_catalog.cc:455] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afee4357bf7845a28ba98ab2842d7ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee4357bf7845a28ba98ab2842d7ec1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37403 } } }
I20260806 23:56:02.231278 4730 sys_catalog.cc:458] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:02.231572 4732 sys_catalog.cc:455] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afee4357bf7845a28ba98ab2842d7ec1. Latest consensus state: current_term: 1 leader_uuid: "afee4357bf7845a28ba98ab2842d7ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afee4357bf7845a28ba98ab2842d7ec1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37403 } } }
I20260806 23:56:02.231689 4732 sys_catalog.cc:458] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:02.231653 4740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:02.235003 4740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:02.240974 4740 catalog_manager.cc:1347] Generated new cluster ID: 4b1d21fa4e25444c89191075a349e140
I20260806 23:56:02.241036 4740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:02.247130 4740 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:02.248064 4740 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:02.252477 4740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1: Generated new TSK 0
I20260806 23:56:02.253137 4740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:02.260345 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37403
--builtin_ntp_servers=127.10.225.148:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 23:56:02.380983 4753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:02.381245 4753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:02.381285 4753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:02.385339 4753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:02.385406 4753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:02.385466 4753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:56:02.390436 4753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:02.391561 4753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:02.392978 4753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:56:02.402436 4753 server_base.cc:1034] running on GCE node
W20260806 23:56:02.402387 4759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:02.402387 4760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:02.402405 4766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:02.403067 4753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:02.403712 4753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:02.404886 4753 hybrid_clock.cc:648] HybridClock initialized: now 1786060562404866 us; error 30 us; skew 500 ppm
I20260806 23:56:02.406708 4753 webserver.cc:460] Webserver started at http://127.10.225.129:37227/ using document root <none> and password file <none>
I20260806 23:56:02.407266 4753 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:02.407342 4753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:02.407546 4753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:02.409731 4753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "9a19d1b6efe24dff847b289b831de6d6"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "44701dadecd45690132f00d25c76c1ea"
server_key_iv: "219a2755e13ea1955cd7a8ccd26e055d"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.410238 4753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "9a19d1b6efe24dff847b289b831de6d6"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "44701dadecd45690132f00d25c76c1ea"
server_key_iv: "219a2755e13ea1955cd7a8ccd26e055d"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.414126 4753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:02.416603 4780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.417884 4753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:02.418016 4753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "9a19d1b6efe24dff847b289b831de6d6"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "44701dadecd45690132f00d25c76c1ea"
server_key_iv: "219a2755e13ea1955cd7a8ccd26e055d"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.418118 4753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:02.440968 4753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:02.441691 4753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:02.441876 4753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:02.442476 4753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:02.443451 4753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:02.443500 4753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.443557 4753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:02.443585 4753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.452292 4753 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:45569
I20260806 23:56:02.452355 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:45569 every 8 connection(s)
I20260806 23:56:02.453439 4753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:56:02.458619 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 4753
I20260806 23:56:02.458727 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:02.459002 11142 external_mini_cluster.cc:1428] Setting key 6e5a3787c6fe7cba39052af8765cebc0
I20260806 23:56:02.460956 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37403
--builtin_ntp_servers=127.10.225.148:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 23:56:02.463095 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.455418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:52055 (local address 127.10.225.190:37403)
0806 23:56:02.455602 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:02.455606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:02.456728 (+ 1122us) server_negotiation.cc:407] Connection header received
0806 23:56:02.458115 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:02.458119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:02.458179 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:02.458254 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:02.459231 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.459873 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.461061 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.461330 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.461425 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.462609 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:02.462612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:02.462614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:02.462626 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:02.462647 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:02.462660 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:02.462719 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:02.462876 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 23:56:02.462945 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 23:56:02.464124 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.455863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:37403 (local address 127.10.225.129:52055)
0806 23:56:02.456571 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:02.456611 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:02.457908 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:02.458448 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:02.458455 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:02.458538 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:02.459091 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.459107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.459993 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.459995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.460935 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.460946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.461482 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.461486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.461648 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.462228 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:56:02.462230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:02.462280 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:56:02.462438 (+ 158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:02.462731 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:02.462742 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:02.462748 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:56:02.462866 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 23:56:02.463177 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":122,"threads_started":1}
I20260806 23:56:02.465579 4948 heartbeater.cc:347] Connected to a master server at 127.10.225.190:37403
I20260806 23:56:02.465819 4948 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:02.466364 4948 heartbeater.cc:511] Master 127.10.225.190:37403 requested a full tablet report, sending...
I20260806 23:56:02.468060 4658 ts_manager.cc:194] Registered new tserver with Master: 9a19d1b6efe24dff847b289b831de6d6 (127.10.225.129:45569)
I20260806 23:56:02.469221 4658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.129:52055
W20260806 23:56:02.578483 4951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:02.578759 4951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:02.578814 4951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:02.582798 4951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:02.582864 4951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:02.582943 4951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:56:02.587751 4951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:02.588863 4951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:02.590335 4951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:02.599737 4961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:02.599730 4963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:02.599730 4960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:02.600097 4951 server_base.cc:1034] running on GCE node
I20260806 23:56:02.600508 4951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:02.601133 4951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:02.602303 4951 hybrid_clock.cc:648] HybridClock initialized: now 1786060562602286 us; error 33 us; skew 500 ppm
I20260806 23:56:02.604190 4951 webserver.cc:460] Webserver started at http://127.10.225.130:44835/ using document root <none> and password file <none>
I20260806 23:56:02.604766 4951 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:02.604835 4951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:02.605031 4951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:02.607079 4951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "428ae86da2854f90999860949cb5966a"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "8284c98600939423eab969d4ec5ad9f4"
server_key_iv: "48d8258ff7284b00e0850e8c2356a69a"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.607626 4951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "428ae86da2854f90999860949cb5966a"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "8284c98600939423eab969d4ec5ad9f4"
server_key_iv: "48d8258ff7284b00e0850e8c2356a69a"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.611378 4951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:56:02.613987 4969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.615216 4951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:02.615339 4951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "428ae86da2854f90999860949cb5966a"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "8284c98600939423eab969d4ec5ad9f4"
server_key_iv: "48d8258ff7284b00e0850e8c2356a69a"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.615453 4951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:02.625466 4951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:02.626129 4951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:02.626300 4951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:02.626869 4951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:02.627835 4951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:02.627883 4951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.627936 4951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:02.627966 4951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.637048 4951 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:44673
I20260806 23:56:02.637074 5155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:44673 every 8 connection(s)
I20260806 23:56:02.637960 4951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:56:02.640146 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 4951
I20260806 23:56:02.640266 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:02.640586 11142 external_mini_cluster.cc:1428] Setting key a8aee3ac2ab9be09c09343fec670f3de
I20260806 23:56:02.642702 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37403
--builtin_ntp_servers=127.10.225.148:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 23:56:02.647609 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.639790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:39259 (local address 127.10.225.190:37403)
0806 23:56:02.639936 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:02.639941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:02.641113 (+ 1172us) server_negotiation.cc:407] Connection header received
0806 23:56:02.642692 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:02.642694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:02.642738 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:02.642819 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:02.643921 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.644527 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.645680 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.645948 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.646044 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.647133 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:02.647138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:02.647141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:02.647157 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:02.647180 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:02.647190 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:02.647266 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:02.647391 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 23:56:02.647452 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:56:02.648463 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.640182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:37403 (local address 127.10.225.130:39259)
0806 23:56:02.640968 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:02.641016 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:02.642485 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:02.642992 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:02.643000 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:02.643099 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:02.643779 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.643798 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.644675 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.644679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.645563 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.645574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.646123 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.646127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.646285 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.646872 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:56:02.646874 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:02.646913 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:56:02.646976 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:02.647265 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:02.647271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:02.647275 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:56:02.647393 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 23:56:02.647625 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260806 23:56:02.649546 5156 heartbeater.cc:347] Connected to a master server at 127.10.225.190:37403
I20260806 23:56:02.649772 5156 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:02.650286 5156 heartbeater.cc:511] Master 127.10.225.190:37403 requested a full tablet report, sending...
I20260806 23:56:02.651271 4658 ts_manager.cc:194] Registered new tserver with Master: 428ae86da2854f90999860949cb5966a (127.10.225.130:44673)
I20260806 23:56:02.651882 4658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.130:39259
W20260806 23:56:02.758060 5160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:02.758462 5160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:02.758553 5160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:02.762656 5160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:02.762714 5160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:02.762797 5160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:56:02.767889 5160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:37403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:02.769021 5160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:02.770615 5160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:02.780624 5169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:02.780582 5173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:02.780607 5171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:02.780797 5160 server_base.cc:1034] running on GCE node
I20260806 23:56:02.781348 5160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:02.781996 5160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:02.783149 5160 hybrid_clock.cc:648] HybridClock initialized: now 1786060562783124 us; error 35 us; skew 500 ppm
I20260806 23:56:02.784997 5160 webserver.cc:460] Webserver started at http://127.10.225.131:41475/ using document root <none> and password file <none>
I20260806 23:56:02.785573 5160 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:02.785634 5160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:02.785840 5160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:02.787954 5160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "830e5f50104d475391586420c78c4c57"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "5802c9db46445375c4d9920ec4d9774b"
server_key_iv: "696860a5ad2c5ceee028f8f0291fb83a"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.788481 5160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "830e5f50104d475391586420c78c4c57"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "5802c9db46445375c4d9920ec4d9774b"
server_key_iv: "696860a5ad2c5ceee028f8f0291fb83a"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.792476 5160 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 23:56:02.795090 5187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.796394 5160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 23:56:02.796518 5160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "830e5f50104d475391586420c78c4c57"
format_stamp: "Formatted at 2026-08-06 23:56:02 on dist-test-slave-nj21"
server_key: "5802c9db46445375c4d9920ec4d9774b"
server_key_iv: "696860a5ad2c5ceee028f8f0291fb83a"
server_key_version: "encryptionkey@0"
I20260806 23:56:02.796622 5160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:02.810149 5160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:02.810895 5160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:02.811080 5160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:02.811766 5160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:02.812772 5160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:02.812822 5160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.812876 5160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:02.812906 5160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:02.822229 5160 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:33779
I20260806 23:56:02.822263 5350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:33779 every 8 connection(s)
I20260806 23:56:02.823195 5160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:56:02.830778 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 5160
I20260806 23:56:02.830926 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:56:02.831202 11142 external_mini_cluster.cc:1428] Setting key 7228e3f16c6e795feef3b824eef35d61
I20260806 23:56:02.832729 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.825102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:34973 (local address 127.10.225.190:37403)
0806 23:56:02.825249 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:02.825254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:02.826343 (+ 1089us) server_negotiation.cc:407] Connection header received
0806 23:56:02.827745 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:02.827748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:02.827801 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:02.827889 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:02.828803 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.829334 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.830504 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.830810 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.830924 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.832166 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:02.832170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:02.832173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:02.832188 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:02.832215 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:02.832226 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:02.832298 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:02.832523 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 23:56:02.832588 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:56:02.833690 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.825483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:37403 (local address 127.10.225.131:34973)
0806 23:56:02.826168 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:02.826207 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:02.827474 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:02.828082 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:02.828089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:02.828171 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:02.828679 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.828691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.829465 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.829468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.830377 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.830387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.831012 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.831017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.831178 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.831866 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:56:02.831868 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:02.831918 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:56:02.831986 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:02.832317 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:02.832326 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:02.832332 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:56:02.832502 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 23:56:02.832777 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":103,"threads_started":1}
I20260806 23:56:02.834784 5356 heartbeater.cc:347] Connected to a master server at 127.10.225.190:37403
I20260806 23:56:02.835034 5356 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:02.835556 5356 heartbeater.cc:511] Master 127.10.225.190:37403 requested a full tablet report, sending...
I20260806 23:56:02.836602 4658 ts_manager.cc:194] Registered new tserver with Master: 830e5f50104d475391586420c78c4c57 (127.10.225.131:33779)
I20260806 23:56:02.837253 4658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.131:34973
I20260806 23:56:02.844995 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:56:02.850519 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.847161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.10.225.190:37403)
0806 23:56:02.847295 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:02.847298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:02.847525 (+ 227us) server_negotiation.cc:407] Connection header received
0806 23:56:02.847620 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:02.847622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:02.847663 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:02.847755 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:02.847968 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.848467 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.849255 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.849469 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.849535 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.850197 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:02.850200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:02.850203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:02.850214 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:02.850233 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:02.850241 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:02.850289 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:02.850353 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 23:56:02.850405 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:56:02.853324 4658 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:56:02.856550 4658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:02.870829 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.865769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.10.225.131:33779)
0806 23:56:02.866424 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:02.866439 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:02.866464 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:56:02.866541 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:02.866546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:02.866739 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:02.867012 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:02.867209 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.868013 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.869442 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.870161 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.870268 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.870456 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:02.870559 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 23:56:02.870657 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":177,"threads_started":1}
I20260806 23:56:02.870820 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.865262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:33779 (local address 127.0.0.1:34732)
0806 23:56:02.865984 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:02.865999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:02.866083 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:02.866886 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:02.866889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:02.866903 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:02.867113 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.867119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.868115 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.868118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.869303 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.869314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.870267 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.870270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.870385 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.870424 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 23:56:02.870531 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 23:56:02.870657 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":73,"thread_start_us":139,"threads_started":1}
I20260806 23:56:02.871090 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.864990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:44673 (local address 127.0.0.1:51758)
0806 23:56:02.865745 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:02.865791 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:02.865936 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:02.866900 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:02.866902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:02.866935 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:02.867097 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.867105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.868641 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.868644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.869671 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.869681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.870691 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.870695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.870785 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.870810 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 23:56:02.870872 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 23:56:02.870944 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":4,"spinlock_wait_cycles":1408,"thread_start_us":75,"threads_started":1}
I20260806 23:56:02.871438 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.865307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45569 (local address 127.0.0.1:47312)
0806 23:56:02.866214 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:02.866227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:02.866309 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:02.866848 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:02.866854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:02.866879 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:02.867118 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.867126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.868608 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.868612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.869778 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:02.869788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.870950 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.870953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:02.871036 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.871057 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 23:56:02.871133 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 23:56:02.871303 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":105,"thread_start_us":101,"threads_started":1}
I20260806 23:56:02.871457 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.865799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.10.225.129:45569)
0806 23:56:02.866245 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:02.866259 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:02.866285 (+ 26us) server_negotiation.cc:407] Connection header received
0806 23:56:02.866465 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:02.866470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:02.866675 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:02.866829 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:02.867247 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.868487 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.869899 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.870629 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.870729 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.870891 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:02.871184 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 23:56:02.871283 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":133,"threads_started":1}
I20260806 23:56:02.871793 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:02.865199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.10.225.130:44673)
0806 23:56:02.866501 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:02.866513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:02.866534 (+ 21us) server_negotiation.cc:407] Connection header received
0806 23:56:02.866602 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:02.866606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:02.866748 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:02.867245 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:02.867392 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.868514 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.869788 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:02.870548 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:02.871286 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:02.871450 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:02.871511 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 23:56:02.871615 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":163,"threads_started":1}
I20260806 23:56:02.874619 5256 tablet_service.cc:1467] Processing CreateTablet for tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b (DEFAULT_TABLE table=test-table [id=eee428c7b335404b9a09b0614d3c2f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:02.874620 4848 tablet_service.cc:1467] Processing CreateTablet for tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b (DEFAULT_TABLE table=test-table [id=eee428c7b335404b9a09b0614d3c2f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:02.875838 4848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:02.875838 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:02.877528 5047 tablet_service.cc:1467] Processing CreateTablet for tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b (DEFAULT_TABLE table=test-table [id=eee428c7b335404b9a09b0614d3c2f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:02.878592 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:02.884539 5390 tablet_bootstrap.cc:492] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6: Bootstrap starting.
I20260806 23:56:02.884845 5392 tablet_bootstrap.cc:492] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: Bootstrap starting.
I20260806 23:56:02.885020 5389 tablet_bootstrap.cc:492] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: Bootstrap starting.
I20260806 23:56:02.887142 5390 tablet_bootstrap.cc:654] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:02.887159 5392 tablet_bootstrap.cc:654] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:02.887159 5389 tablet_bootstrap.cc:654] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:02.887998 5390 log.cc:824] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:02.888036 5392 log.cc:824] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:02.888036 5389 log.cc:824] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:02.890185 5389 tablet_bootstrap.cc:492] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: No bootstrap required, opened a new log
I20260806 23:56:02.890196 5390 tablet_bootstrap.cc:492] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6: No bootstrap required, opened a new log
I20260806 23:56:02.890401 5390 ts_tablet_manager.cc:1397] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:56:02.890403 5389 ts_tablet_manager.cc:1397] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:56:02.890457 5392 tablet_bootstrap.cc:492] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: No bootstrap required, opened a new log
I20260806 23:56:02.890654 5392 ts_tablet_manager.cc:1397] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:56:02.893790 5390 raft_consensus.cc:348] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } }
I20260806 23:56:02.894026 5390 raft_consensus.cc:374] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:02.894097 5390 raft_consensus.cc:729] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a19d1b6efe24dff847b289b831de6d6, State: Initialized, Role: FOLLOWER
I20260806 23:56:02.894562 5390 consensus_queue.cc:260] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } }
I20260806 23:56:02.894695 5392 raft_consensus.cc:348] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } }
I20260806 23:56:02.894692 5389 raft_consensus.cc:348] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } }
I20260806 23:56:02.894940 5389 raft_consensus.cc:374] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:02.895011 5392 raft_consensus.cc:374] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:02.895032 5389 raft_consensus.cc:729] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830e5f50104d475391586420c78c4c57, State: Initialized, Role: FOLLOWER
I20260806 23:56:02.895092 5392 raft_consensus.cc:729] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428ae86da2854f90999860949cb5966a, State: Initialized, Role: FOLLOWER
I20260806 23:56:02.895267 4948 heartbeater.cc:503] Master 127.10.225.190:37403 was elected leader, sending a full tablet report...
I20260806 23:56:02.895510 5392 consensus_queue.cc:260] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } }
I20260806 23:56:02.895598 5390 ts_tablet_manager.cc:1428] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:56:02.896159 5389 consensus_queue.cc:260] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } }
I20260806 23:56:02.896528 5392 ts_tablet_manager.cc:1428] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 23:56:02.896600 5156 heartbeater.cc:503] Master 127.10.225.190:37403 was elected leader, sending a full tablet report...
I20260806 23:56:02.897147 5389 ts_tablet_manager.cc:1428] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 23:56:02.897629 5356 heartbeater.cc:503] Master 127.10.225.190:37403 was elected leader, sending a full tablet report...
W20260806 23:56:02.954923 4949 tablet.cc:2369] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:56:03.074250 5359 tablet.cc:2369] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:56:03.139120 5157 tablet.cc:2369] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:03.155244 5399 raft_consensus.cc:482] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:03.155491 5399 raft_consensus.cc:504] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } }
I20260806 23:56:03.156726 5399 leader_election.cc:288] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428ae86da2854f90999860949cb5966a (127.10.225.130:44673), 9a19d1b6efe24dff847b289b831de6d6 (127.10.225.129:45569)
I20260806 23:56:03.160640 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:03.157120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:59137 (local address 127.10.225.129:45569)
0806 23:56:03.157283 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:03.157288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:03.157307 (+ 19us) server_negotiation.cc:407] Connection header received
0806 23:56:03.157432 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:03.157436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:03.157490 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:03.157562 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:03.157863 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.158521 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.159698 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.160112 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.160224 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:03.160278 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:03.160454 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 23:56:03.160528 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 23:56:03.160640 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:03.157016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45569 (local address 127.10.225.131:59137)
0806 23:56:03.157145 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:03.157162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:03.157292 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:03.157594 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:03.157597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:03.157622 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:03.157748 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:03.157756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.158621 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.158632 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:03.159528 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:03.159540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.160224 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.160228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:03.160336 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:03.160356 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 23:56:03.160423 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 23:56:03.160501 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260806 23:56:03.160797 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:03.157158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:43967 (local address 127.10.225.130:44673)
0806 23:56:03.157420 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:03.157424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:03.157726 (+ 302us) server_negotiation.cc:407] Connection header received
0806 23:56:03.157828 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:03.157830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:03.157881 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:03.157959 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:03.158228 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.158845 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.159895 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.160275 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.160392 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:03.160441 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:03.160614 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 23:56:03.160685 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260806 23:56:03.160818 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:03.157016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:44673 (local address 127.10.225.131:43967)
0806 23:56:03.157624 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:03.157637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:03.157734 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:03.157977 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:03.157980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:03.157999 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:03.158120 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:03.158126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.158967 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.158971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:03.159768 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:03.159788 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.160409 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.160413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:03.160504 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:03.160522 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 23:56:03.160585 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 23:56:03.160651 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":191,"thread_start_us":131,"threads_started":1}
I20260806 23:56:03.161394 4877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a4a972b4984cc2b8e7b55c2ef7bf6b" candidate_uuid: "830e5f50104d475391586420c78c4c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a19d1b6efe24dff847b289b831de6d6" is_pre_election: true
I20260806 23:56:03.161394 5079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a4a972b4984cc2b8e7b55c2ef7bf6b" candidate_uuid: "830e5f50104d475391586420c78c4c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428ae86da2854f90999860949cb5966a" is_pre_election: true
I20260806 23:56:03.161741 5079 raft_consensus.cc:2454] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830e5f50104d475391586420c78c4c57 in term 0.
I20260806 23:56:03.161741 4877 raft_consensus.cc:2454] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830e5f50104d475391586420c78c4c57 in term 0.
I20260806 23:56:03.162251 5190 leader_election.cc:302] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428ae86da2854f90999860949cb5966a, 830e5f50104d475391586420c78c4c57; no voters:
I20260806 23:56:03.162542 5399 raft_consensus.cc:2781] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:03.162633 5399 raft_consensus.cc:482] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:03.162680 5399 raft_consensus.cc:3037] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:03.163746 5399 raft_consensus.cc:504] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } }
I20260806 23:56:03.164090 5399 leader_election.cc:288] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [CANDIDATE]: Term 1 election: Requested vote from peers 428ae86da2854f90999860949cb5966a (127.10.225.130:44673), 9a19d1b6efe24dff847b289b831de6d6 (127.10.225.129:45569)
I20260806 23:56:03.164505 5079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a4a972b4984cc2b8e7b55c2ef7bf6b" candidate_uuid: "830e5f50104d475391586420c78c4c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428ae86da2854f90999860949cb5966a"
I20260806 23:56:03.164505 4877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a4a972b4984cc2b8e7b55c2ef7bf6b" candidate_uuid: "830e5f50104d475391586420c78c4c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a19d1b6efe24dff847b289b831de6d6"
I20260806 23:56:03.164642 4877 raft_consensus.cc:3037] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:03.164642 5079 raft_consensus.cc:3037] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:03.165962 4877 raft_consensus.cc:2454] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830e5f50104d475391586420c78c4c57 in term 1.
I20260806 23:56:03.165985 5079 raft_consensus.cc:2454] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830e5f50104d475391586420c78c4c57 in term 1.
I20260806 23:56:03.166271 5190 leader_election.cc:302] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428ae86da2854f90999860949cb5966a, 830e5f50104d475391586420c78c4c57; no voters:
I20260806 23:56:03.166500 5399 raft_consensus.cc:2781] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:03.166707 5399 raft_consensus.cc:686] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 1 LEADER]: Becoming Leader. State: Replica: 830e5f50104d475391586420c78c4c57, State: Running, Role: LEADER
I20260806 23:56:03.167023 5399 consensus_queue.cc:237] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } }
I20260806 23:56:03.170686 4658 catalog_manager.cc:5495] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830e5f50104d475391586420c78c4c57 (127.10.225.131). New cstate: current_term: 1 leader_uuid: "830e5f50104d475391586420c78c4c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830e5f50104d475391586420c78c4c57" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 33779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 } health_report { overall_health: UNKNOWN } } }
I20260806 23:56:03.237700 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:03.234290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.10.225.131:33779)
0806 23:56:03.234422 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:03.234426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:03.234441 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:56:03.234530 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:03.234532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:03.234591 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:03.234682 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:03.234914 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.235419 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.236272 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:03.236488 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:03.236574 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:03.236870 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:03.237295 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:03.237404 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:03.237476 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 23:56:03.237537 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:56:03.243532 4877 raft_consensus.cc:1261] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 9a19d1b6efe24dff847b289b831de6d6 [term 1 FOLLOWER]: Refusing update from remote peer 830e5f50104d475391586420c78c4c57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:03.243582 5079 raft_consensus.cc:1261] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [term 1 FOLLOWER]: Refusing update from remote peer 830e5f50104d475391586420c78c4c57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:03.244302 5401 consensus_queue.cc:1035] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428ae86da2854f90999860949cb5966a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 44673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:03.244485 5399 consensus_queue.cc:1035] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a19d1b6efe24dff847b289b831de6d6" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:03.252665 5408 mvcc.cc:205] Tried to move back new op lower bound from 7315704067038101504 to 7315704066735276032. Current Snapshot: MvccSnapshot[applied={T|T < 7315704067038101504}]
I20260806 23:56:03.256042 5409 mvcc.cc:205] Tried to move back new op lower bound from 7315704067038101504 to 7315704066735276032. Current Snapshot: MvccSnapshot[applied={T|T < 7315704067038101504 or (T in {7315704067038101504})}]
I20260806 23:56:03.261279 4659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:03.261493 4659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:03.264290 4659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afee4357bf7845a28ba98ab2842d7ec1: Sending DeleteTablet for 3 replicas of tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b
I20260806 23:56:03.265198 5256 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:03 UTC) from {username='slave'} at 127.0.0.1:34732
I20260806 23:56:03.265208 5047 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:03 UTC) from {username='slave'} at 127.0.0.1:51758
I20260806 23:56:03.265808 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 4753
I20260806 23:56:03.272799 5420 tablet_replica.cc:330] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: stopping tablet replica
I20260806 23:56:03.273252 5420 raft_consensus.cc:2229] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:56:03.273761 5420 raft_consensus.cc:2258] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:03.274248 5421 tablet_replica.cc:330] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: stopping tablet replica
I20260806 23:56:03.274600 5421 raft_consensus.cc:2229] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:03.275015 5421 raft_consensus.cc:2258] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:03.275717 5420 ts_tablet_manager.cc:1905] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:03.276517 5421 ts_tablet_manager.cc:1905] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:03.278584 5420 ts_tablet_manager.cc:1918] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:03.278764 5420 log.cc:1196] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/b5a4a972b4984cc2b8e7b55c2ef7bf6b
I20260806 23:56:03.279163 5420 ts_tablet_manager.cc:1939] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 830e5f50104d475391586420c78c4c57: Deleting consensus metadata
I20260806 23:56:03.279168 5421 ts_tablet_manager.cc:1918] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:03.279248 5421 log.cc:1196] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/b5a4a972b4984cc2b8e7b55c2ef7bf6b
I20260806 23:56:03.279608 5421 ts_tablet_manager.cc:1939] T b5a4a972b4984cc2b8e7b55c2ef7bf6b P 428ae86da2854f90999860949cb5966a: Deleting consensus metadata
I20260806 23:56:03.280640 4638 catalog_manager.cc:4841] TS 830e5f50104d475391586420c78c4c57 (127.10.225.131:33779): tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b (table test-table [id=eee428c7b335404b9a09b0614d3c2f0f]) successfully deleted
I20260806 23:56:03.281041 4635 catalog_manager.cc:4841] TS 428ae86da2854f90999860949cb5966a (127.10.225.130:44673): tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b (table test-table [id=eee428c7b335404b9a09b0614d3c2f0f]) successfully deleted
W20260806 23:56:03.281704 4632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:45569 (error 108)
I20260806 23:56:03.282478 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 4951
I20260806 23:56:03.282728 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:03.282458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45569 (local address 127.0.0.1:47328)
0806 23:56:03.282583 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:03.282647 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 23:56:03.296481 4632 catalog_manager.cc:4584] TS 9a19d1b6efe24dff847b289b831de6d6 (127.10.225.129:45569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5a4a972b4984cc2b8e7b55c2ef7bf6b: Network error: Client connection negotiation failed: client connection to 127.10.225.129:45569: connect: Connection refused (error 111)
I20260806 23:56:03.302779 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 5160
I20260806 23:56:03.317546 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 4607
2026-08-06T23:56:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1321 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:03 dist-test-slave-nj21 krb5kdc[5426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:03 dist-test-slave-nj21 krb5kdc[5426](info): set up 2 sockets
Aug 06 23:56:03 dist-test-slave-nj21 krb5kdc[5426](info): commencing operation
krb5kdc: starting...
W20260806 23:56:04.456750 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:04 dist-test-slave-nj21 krb5kdc[5426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:04.582454 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:44369
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:04.696707 5445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:04.697007 5445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:04.697050 5445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:04.701570 5445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:04.701634 5445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:04.701653 5445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:04.701669 5445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:04.701684 5445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:04.706977 5445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:44369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:04.708039 5445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:04.709031 5445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:04.717257 5450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:04.717309 5453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:04.717262 5458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:04.717621 5445 server_base.cc:1034] running on GCE node
I20260806 23:56:04.718175 5445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:04.719323 5445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:04.720484 5445 hybrid_clock.cc:648] HybridClock initialized: now 1786060564720471 us; error 27 us; skew 500 ppm
Aug 06 23:56:04 dist-test-slave-nj21 krb5kdc[5426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060564, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:04.726099 5445 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:04.727110 5445 webserver.cc:460] Webserver started at http://127.10.225.190:35189/ using document root <none> and password file <none>
I20260806 23:56:04.727723 5445 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:04.727806 5445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:04.728019 5445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:04.730134 5445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "48b29520b4a244afbf7bc62b325ed61c"
format_stamp: "Formatted at 2026-08-06 23:56:04 on dist-test-slave-nj21"
server_key: "2fe42ab3dabbdba5979214eb7175726c"
server_key_iv: "98897cf3814f587df3677f954b647ccb"
server_key_version: "encryptionkey@0"
I20260806 23:56:04.730690 5445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "48b29520b4a244afbf7bc62b325ed61c"
format_stamp: "Formatted at 2026-08-06 23:56:04 on dist-test-slave-nj21"
server_key: "2fe42ab3dabbdba5979214eb7175726c"
server_key_iv: "98897cf3814f587df3677f954b647ccb"
server_key_version: "encryptionkey@0"
I20260806 23:56:04.734459 5445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:56:04.736972 5470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:04.738153 5445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:56:04.738279 5445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "48b29520b4a244afbf7bc62b325ed61c"
format_stamp: "Formatted at 2026-08-06 23:56:04 on dist-test-slave-nj21"
server_key: "2fe42ab3dabbdba5979214eb7175726c"
server_key_iv: "98897cf3814f587df3677f954b647ccb"
server_key_version: "encryptionkey@0"
I20260806 23:56:04.738392 5445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:04.757230 5445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:04.760385 5445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:04.760576 5445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:04.767187 5445 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:44369
I20260806 23:56:04.767251 5546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:44369 every 8 connection(s)
I20260806 23:56:04.768177 5445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:04.769790 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 5445
I20260806 23:56:04.769905 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:04.770195 11142 external_mini_cluster.cc:1428] Setting key 05ce0099f091f18fbdb83ec15b5f5846
I20260806 23:56:04.771134 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:04.778242 5547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c: Bootstrap starting.
Aug 06 23:56:04 dist-test-slave-nj21 krb5kdc[5426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:04.781030 5547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:04.781978 5547 log.cc:824] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:04.784262 5547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c: No bootstrap required, opened a new log
I20260806 23:56:04.785159 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:04.771529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.10.225.190:44369)
0806 23:56:04.771982 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:04.772002 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:04.772044 (+ 42us) server_negotiation.cc:407] Connection header received
0806 23:56:04.772856 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:04.772891 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:04.773289 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:04.773627 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:04.774329 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:04.775801 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:04.776655 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:04.777479 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:04.777665 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:04.780521 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:04.780527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:04.780540 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:04.780569 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:04.782467 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:04.782916 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:04.782921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:04.782926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:04.783040 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:04.783296 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:04.783298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:04.783299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:04.783720 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:04.783875 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:04.784135 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 23:56:04.784404 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":154,"threads_started":1}
I20260806 23:56:04.787875 5547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b29520b4a244afbf7bc62b325ed61c" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44369 } }
I20260806 23:56:04.788113 5547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:04.788151 5547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b29520b4a244afbf7bc62b325ed61c, State: Initialized, Role: FOLLOWER
I20260806 23:56:04.788558 5547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b29520b4a244afbf7bc62b325ed61c" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44369 } }
I20260806 23:56:04.788683 5547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:04.788753 5547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:04.788851 5547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:04.789882 5547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b29520b4a244afbf7bc62b325ed61c" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44369 } }
I20260806 23:56:04.790200 5547 leader_election.cc:302] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b29520b4a244afbf7bc62b325ed61c; no voters:
I20260806 23:56:04.790530 5547 leader_election.cc:288] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:04.790678 5556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:04.790933 5556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [term 1 LEADER]: Becoming Leader. State: Replica: 48b29520b4a244afbf7bc62b325ed61c, State: Running, Role: LEADER
I20260806 23:56:04.791232 5556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b29520b4a244afbf7bc62b325ed61c" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44369 } }
I20260806 23:56:04.791741 5547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:04.792845 5558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b29520b4a244afbf7bc62b325ed61c. Latest consensus state: current_term: 1 leader_uuid: "48b29520b4a244afbf7bc62b325ed61c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b29520b4a244afbf7bc62b325ed61c" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44369 } } }
I20260806 23:56:04.792979 5558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:04.793326 5557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48b29520b4a244afbf7bc62b325ed61c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b29520b4a244afbf7bc62b325ed61c" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 44369 } } }
I20260806 23:56:04.793478 5557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:04.793388 5571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:04.796880 5571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:04.803820 5571 catalog_manager.cc:1347] Generated new cluster ID: 6745d4dcb74b4099a52c9050eadbf26d
I20260806 23:56:04.803988 5571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:04.833189 5571 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:04.834261 5571 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:04.843431 5571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c: Generated new TSK 0
I20260806 23:56:04.844182 5571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:56:04.880903 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:44369
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:05.000576 5586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:05.000857 5586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:05.000921 5586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:05.004946 5586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:05.005010 5586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:05.005091 5586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:56:05.010710 5586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:05.011933 5586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:05.012998 5586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:05.022675 5595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:05.022676 5592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:05.022735 5593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:05.022897 5586 server_base.cc:1034] running on GCE node
I20260806 23:56:05.023334 5586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:05.024009 5586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:05.025166 5586 hybrid_clock.cc:648] HybridClock initialized: now 1786060565025148 us; error 32 us; skew 500 ppm
Aug 06 23:56:05 dist-test-slave-nj21 krb5kdc[5426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060565, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:05.027776 5586 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:56:05.028621 5586 webserver.cc:460] Webserver started at http://127.10.225.129:35907/ using document root <none> and password file <none>
I20260806 23:56:05.029179 5586 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:05.029243 5586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:05.029451 5586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:05.031529 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "4ddd53e97b904d65837e14ac8d02de41"
format_stamp: "Formatted at 2026-08-06 23:56:05 on dist-test-slave-nj21"
server_key: "a7407ea09fef8985ae5d080e0ab7439f"
server_key_iv: "9f20de2a371672e5c90f0d56a36263a2"
server_key_version: "encryptionkey@0"
I20260806 23:56:05.032120 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "4ddd53e97b904d65837e14ac8d02de41"
format_stamp: "Formatted at 2026-08-06 23:56:05 on dist-test-slave-nj21"
server_key: "a7407ea09fef8985ae5d080e0ab7439f"
server_key_iv: "9f20de2a371672e5c90f0d56a36263a2"
server_key_version: "encryptionkey@0"
I20260806 23:56:05.036185 5586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:56:05.038663 5614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:05.040068 5586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:05.040186 5586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "4ddd53e97b904d65837e14ac8d02de41"
format_stamp: "Formatted at 2026-08-06 23:56:05 on dist-test-slave-nj21"
server_key: "a7407ea09fef8985ae5d080e0ab7439f"
server_key_iv: "9f20de2a371672e5c90f0d56a36263a2"
server_key_version: "encryptionkey@0"
I20260806 23:56:05.040284 5586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:05.057757 5586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:05.061153 5586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:05.061348 5586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:05.061977 5586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:05.062858 5586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:05.062916 5586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:05.062971 5586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:05.062999 5586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:05.071830 5586 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:41265
I20260806 23:56:05.071903 5782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:41265 every 8 connection(s)
I20260806 23:56:05.072758 5586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:56:05.078647 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 5586
I20260806 23:56:05.078778 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:05.079066 11142 external_mini_cluster.cc:1428] Setting key 8d6a548ab5c5a3af84772224209d69b5
Aug 06 23:56:05 dist-test-slave-nj21 krb5kdc[5426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060565, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:05.086750 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.074579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:46009 (local address 127.10.225.190:44369)
0806 23:56:05.074750 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.074755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.075574 (+ 819us) server_negotiation.cc:407] Connection header received
0806 23:56:05.076925 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.076928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.076982 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.077066 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:05.078198 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.078885 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.080402 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.080697 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.080805 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.083411 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:05.083414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:05.083416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:05.083440 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:05.084778 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:05.085259 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:05.085262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:05.085263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:05.085341 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:05.085653 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:05.085656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:05.085658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:05.085852 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:05.085938 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.086441 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.086575 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 23:56:05.087551 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.074950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:44369 (local address 127.10.225.129:46009)
0806 23:56:05.075425 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.075468 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:05.076702 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:05.077205 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:05.077211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:05.077552 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:05.078044 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.078058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.079026 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.079029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.080255 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.080269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.080853 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.080857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.081009 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.081638 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:05.081656 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:05.083248 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:05.084896 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:05.084902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:05.084918 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:05.085167 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:05.085426 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:05.085429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:05.085431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:05.085568 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:05.085952 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:05.085959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:05.086205 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 23:56:05.086383 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 23:56:05.086714 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260806 23:56:05.089016 5783 heartbeater.cc:347] Connected to a master server at 127.10.225.190:44369
I20260806 23:56:05.089242 5783 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:56:05.089749 5783 heartbeater.cc:511] Master 127.10.225.190:44369 requested a full tablet report, sending...
I20260806 23:56:05.091404 5496 ts_manager.cc:194] Registered new tserver with Master: 4ddd53e97b904d65837e14ac8d02de41 (127.10.225.129:41265)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:56:05.092880 5496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:46009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:56:05.107869 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:44369
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:05.224392 5792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:05.224660 5792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:05.224705 5792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:05.228745 5792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:05.228819 5792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:05.228896 5792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:56:05.234412 5792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:05.235492 5792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:05.236580 5792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:05.246163 5806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:05.246167 5803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:05.246168 5802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:05.246577 5792 server_base.cc:1034] running on GCE node
I20260806 23:56:05.246979 5792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:05.247648 5792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:05.248832 5792 hybrid_clock.cc:648] HybridClock initialized: now 1786060565248793 us; error 49 us; skew 500 ppm
Aug 06 23:56:05 dist-test-slave-nj21 krb5kdc[5426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060565, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:05.251560 5792 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:56:05.252480 5792 webserver.cc:460] Webserver started at http://127.10.225.130:33349/ using document root <none> and password file <none>
I20260806 23:56:05.253021 5792 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:05.253084 5792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:05.253298 5792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:05.255371 5792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "a481c0ad145846ea86ac1152f768c25c"
format_stamp: "Formatted at 2026-08-06 23:56:05 on dist-test-slave-nj21"
server_key: "6274aed449a7cb4132f9219fdeaf976b"
server_key_iv: "ad4d2f4fef89ccff29df7c4b0309edad"
server_key_version: "encryptionkey@0"
I20260806 23:56:05.255925 5792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "a481c0ad145846ea86ac1152f768c25c"
format_stamp: "Formatted at 2026-08-06 23:56:05 on dist-test-slave-nj21"
server_key: "6274aed449a7cb4132f9219fdeaf976b"
server_key_iv: "ad4d2f4fef89ccff29df7c4b0309edad"
server_key_version: "encryptionkey@0"
I20260806 23:56:05.259490 5792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:56:05.261978 5820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:05.263161 5792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:05.263263 5792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "a481c0ad145846ea86ac1152f768c25c"
format_stamp: "Formatted at 2026-08-06 23:56:05 on dist-test-slave-nj21"
server_key: "6274aed449a7cb4132f9219fdeaf976b"
server_key_iv: "ad4d2f4fef89ccff29df7c4b0309edad"
server_key_version: "encryptionkey@0"
I20260806 23:56:05.263381 5792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:05.277362 5792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:05.280259 5792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:05.280450 5792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:05.281050 5792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:05.281905 5792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:05.281952 5792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:05.281997 5792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:05.282025 5792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:05.290889 5792 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:35195
I20260806 23:56:05.290910 5995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:35195 every 8 connection(s)
I20260806 23:56:05.291844 5792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:56:05.295542 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 5792
I20260806 23:56:05.295696 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:05.296021 11142 external_mini_cluster.cc:1428] Setting key 485e84fe638de16b18d30bb5f485bd41
Aug 06 23:56:05 dist-test-slave-nj21 krb5kdc[5426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060565, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
I20260806 23:56:05.306762 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.293766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:39389 (local address 127.10.225.190:44369)
0806 23:56:05.293907 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.293912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.294758 (+ 846us) server_negotiation.cc:407] Connection header received
0806 23:56:05.296365 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.296369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.296443 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.296540 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:05.297879 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.298521 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.300276 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.300508 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.300600 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.303179 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:05.303183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:05.303187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:05.303213 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:05.304563 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:05.305056 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:05.305060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:05.305061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:05.305140 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:05.305459 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:05.305462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:05.305464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:05.305662 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:05.305750 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.306440 (+ 690us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.306586 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:56:05.307660 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.294161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:44369 (local address 127.10.225.130:39389)
0806 23:56:05.294609 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.294653 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:05.296111 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:05.296691 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:05.296699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:05.297088 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:05.297714 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.297736 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.298668 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.298672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.300131 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.300142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.300657 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.300661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.300831 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.301496 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:05.301514 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:05.303010 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:05.304670 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:05.304677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:05.304693 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:05.304963 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:05.305223 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:05.305226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:05.305228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:05.305369 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:05.305760 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:05.305767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:05.306175 (+ 408us) client_negotiation.cc:769] Sending connection context
0806 23:56:05.306387 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 23:56:05.306742 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
I20260806 23:56:05.308869 5996 heartbeater.cc:347] Connected to a master server at 127.10.225.190:44369
I20260806 23:56:05.309150 5996 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:05.309772 5996 heartbeater.cc:511] Master 127.10.225.190:44369 requested a full tablet report, sending...
I20260806 23:56:05.310925 5496 ts_manager.cc:194] Registered new tserver with Master: a481c0ad145846ea86ac1152f768c25c (127.10.225.130:35195)
I20260806 23:56:05.311529 5496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:39389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:56:05.324704 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:44369
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:05.441494 6004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:05.441787 6004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:05.441833 6004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:05.446092 6004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:05.446174 6004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:05.446269 6004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:56:05.451987 6004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:05.453048 6004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:05.454092 6004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:05.463644 6015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:05.463786 6004 server_base.cc:1034] running on GCE node
W20260806 23:56:05.463649 6010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:05.463663 6019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:05.464411 6004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:05.465054 6004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:05.466241 6004 hybrid_clock.cc:648] HybridClock initialized: now 1786060565466223 us; error 24 us; skew 500 ppm
Aug 06 23:56:05 dist-test-slave-nj21 krb5kdc[5426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060565, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:05.469038 6004 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:56:05.469951 6004 webserver.cc:460] Webserver started at http://127.10.225.131:36009/ using document root <none> and password file <none>
I20260806 23:56:05.470504 6004 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:05.470571 6004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:05.470784 6004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:05.472911 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "36082c29f0d64b0da6497c3f9bb24b14"
format_stamp: "Formatted at 2026-08-06 23:56:05 on dist-test-slave-nj21"
server_key: "11234424e4e14c5cc6a52be0f1eb375c"
server_key_iv: "5859440da99ae2c1291a53a0503ac647"
server_key_version: "encryptionkey@0"
I20260806 23:56:05.473480 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "36082c29f0d64b0da6497c3f9bb24b14"
format_stamp: "Formatted at 2026-08-06 23:56:05 on dist-test-slave-nj21"
server_key: "11234424e4e14c5cc6a52be0f1eb375c"
server_key_iv: "5859440da99ae2c1291a53a0503ac647"
server_key_version: "encryptionkey@0"
I20260806 23:56:05.477351 6004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:05.479946 6028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:05.481221 6004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:05.481338 6004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "36082c29f0d64b0da6497c3f9bb24b14"
format_stamp: "Formatted at 2026-08-06 23:56:05 on dist-test-slave-nj21"
server_key: "11234424e4e14c5cc6a52be0f1eb375c"
server_key_iv: "5859440da99ae2c1291a53a0503ac647"
server_key_version: "encryptionkey@0"
I20260806 23:56:05.481438 6004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:05.505518 6004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:05.508852 6004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:05.509040 6004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:05.509680 6004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:05.510571 6004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:05.510620 6004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:05.510676 6004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:05.510701 6004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:05.519580 6004 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:35143
I20260806 23:56:05.519603 6213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:35143 every 8 connection(s)
I20260806 23:56:05.520582 6004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:56:05.522820 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 6004
I20260806 23:56:05.522957 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:56:05.523232 11142 external_mini_cluster.cc:1428] Setting key 3b096e0ececb6676ec8f01cadbc11d76
Aug 06 23:56:05 dist-test-slave-nj21 krb5kdc[5426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060565, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:05.535410 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.522506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:54453 (local address 127.10.225.190:44369)
0806 23:56:05.522689 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.522694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.523591 (+ 897us) server_negotiation.cc:407] Connection header received
0806 23:56:05.525229 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.525232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.525279 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.525350 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:05.526580 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.527129 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.528406 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.528627 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.528712 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.531297 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:05.531300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:05.531303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:05.531331 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:05.532709 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:05.533315 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:05.533320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:05.533321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:05.533592 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:05.534214 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:05.534217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:05.534219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:05.534443 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:05.534530 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.535148 (+ 618us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.535271 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 23:56:05.536427 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.522910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:44369 (local address 127.10.225.131:54453)
0806 23:56:05.523429 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.523484 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:05.524998 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:05.525540 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:05.525549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:05.525934 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:05.526441 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.526456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.527279 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.527282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.528272 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.528283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.528762 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.528767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.528921 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.529558 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:05.529585 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:05.531123 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:05.532818 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:05.532823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:05.532836 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:05.533119 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:05.533775 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:05.533779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:05.533789 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:05.534075 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:05.534590 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:05.534596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:05.534888 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 23:56:05.535079 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 23:56:05.535376 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
I20260806 23:56:05.537554 6214 heartbeater.cc:347] Connected to a master server at 127.10.225.190:44369
I20260806 23:56:05.537820 6214 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:05.538331 6214 heartbeater.cc:511] Master 127.10.225.190:44369 requested a full tablet report, sending...
I20260806 23:56:05.539381 5496 ts_manager.cc:194] Registered new tserver with Master: 36082c29f0d64b0da6497c3f9bb24b14 (127.10.225.131:35143)
I20260806 23:56:05.539996 5496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:54453
I20260806 23:56:05.549744 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:56:05.559554 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.552006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45200 (local address 127.10.225.190:44369)
0806 23:56:05.552152 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.552155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.552258 (+ 103us) server_negotiation.cc:407] Connection header received
0806 23:56:05.552359 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.552362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.552405 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.552473 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:05.553152 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.553671 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.554510 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.554723 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.554799 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.556418 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:05.556422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:05.556424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:05.556448 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:05.557934 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:05.558458 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:05.558461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:05.558462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:05.558542 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:05.558781 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:05.558783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:05.558784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:05.559008 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:05.559119 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.559345 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.559425 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 23:56:05.562587 5496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:56:05.565881 5496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:05.582531 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.576824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.10.225.131:35143)
0806 23:56:05.577741 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.577754 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.577788 (+ 34us) server_negotiation.cc:407] Connection header received
0806 23:56:05.577864 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.577869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.578038 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.578543 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:05.578788 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.579986 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.581226 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.581833 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.581965 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.582115 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.582262 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.582363 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":218,"threads_started":1}
I20260806 23:56:05.582520 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.575740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:35143 (local address 127.0.0.1:38544)
0806 23:56:05.576750 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.576821 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:05.576962 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:05.578197 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:05.578200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:05.578218 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:05.578530 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.578541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.580113 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.580117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.581095 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.581106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.581960 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.581964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.582074 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.582113 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 23:56:05.582223 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 23:56:05.582363 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":60,"thread_start_us":133,"threads_started":1}
I20260806 23:56:05.582726 6230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.575906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35195 (local address 127.0.0.1:35906)
0806 23:56:05.577372 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.577390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:05.577518 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:05.578373 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:05.578376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:05.578392 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:05.578568 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.578574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.579541 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.579545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.580958 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.580970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.582266 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.582269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.582402 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.582426 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:56:05.582494 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 23:56:05.582585 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"mutex_wait_us":40,"thread_start_us":90,"threads_started":1}
I20260806 23:56:05.582820 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.575929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.10.225.129:41265)
0806 23:56:05.576796 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.576817 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.577063 (+ 246us) server_negotiation.cc:407] Connection header received
0806 23:56:05.577305 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.577312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.577551 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.577684 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:05.578235 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.579758 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.581138 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.581957 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.582075 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.582197 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.582582 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.582679 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":147,"threads_started":1}
I20260806 23:56:05.583169 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.575791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:41265 (local address 127.0.0.1:40902)
0806 23:56:05.576952 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.576966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:05.577129 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:05.577910 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:05.577916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:05.577943 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:05.578107 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.578116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.579906 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.579909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.581021 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.581029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.582263 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.582266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.582412 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.582429 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 23:56:05.582494 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:56:05.583054 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":508,"thread_start_us":249,"threads_started":1}
I20260806 23:56:05.583201 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.576810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.10.225.130:35195)
0806 23:56:05.577428 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.577441 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.577470 (+ 29us) server_negotiation.cc:407] Connection header received
0806 23:56:05.577829 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.577835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.578015 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.578174 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:05.578678 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.579418 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.581099 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.581957 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.582075 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.582197 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.582941 (+ 744us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.583058 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":146,"threads_started":1}
I20260806 23:56:05.586189 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 2c772505eb554b11a749663c39123296 (DEFAULT_TABLE table=test-table [id=b835bb62e22b4d0fa0d877ab9bd2b86a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:05.586593 6113 tablet_service.cc:1467] Processing CreateTablet for tablet 2c772505eb554b11a749663c39123296 (DEFAULT_TABLE table=test-table [id=b835bb62e22b4d0fa0d877ab9bd2b86a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:05.586592 5897 tablet_service.cc:1467] Processing CreateTablet for tablet 2c772505eb554b11a749663c39123296 (DEFAULT_TABLE table=test-table [id=b835bb62e22b4d0fa0d877ab9bd2b86a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:05.587253 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c772505eb554b11a749663c39123296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:05.587631 6113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c772505eb554b11a749663c39123296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:05.587631 5897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c772505eb554b11a749663c39123296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:05.595283 6240 tablet_bootstrap.cc:492] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: Bootstrap starting.
I20260806 23:56:05.596036 6243 tablet_bootstrap.cc:492] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: Bootstrap starting.
I20260806 23:56:05.596668 6244 tablet_bootstrap.cc:492] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41: Bootstrap starting.
I20260806 23:56:05.598512 6244 tablet_bootstrap.cc:654] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:05.598644 6240 tablet_bootstrap.cc:654] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:05.598645 6243 tablet_bootstrap.cc:654] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:05.599342 6244 log.cc:824] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:05.600131 6243 log.cc:824] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:05.600131 6240 log.cc:824] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:05.601732 6244 tablet_bootstrap.cc:492] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41: No bootstrap required, opened a new log
I20260806 23:56:05.601909 6240 tablet_bootstrap.cc:492] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: No bootstrap required, opened a new log
I20260806 23:56:05.601967 6244 ts_tablet_manager.cc:1397] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 23:56:05.602097 6240 ts_tablet_manager.cc:1397] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 23:56:05.602133 6243 tablet_bootstrap.cc:492] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: No bootstrap required, opened a new log
I20260806 23:56:05.602363 6243 ts_tablet_manager.cc:1397] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 23:56:05.605165 6240 raft_consensus.cc:348] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } }
I20260806 23:56:05.605409 6240 raft_consensus.cc:374] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:05.605482 6240 raft_consensus.cc:729] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36082c29f0d64b0da6497c3f9bb24b14, State: Initialized, Role: FOLLOWER
I20260806 23:56:05.605933 6240 consensus_queue.cc:260] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } }
I20260806 23:56:05.606122 6243 raft_consensus.cc:348] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } }
I20260806 23:56:05.606446 6244 raft_consensus.cc:348] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } }
I20260806 23:56:05.606717 6244 raft_consensus.cc:374] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:05.606731 6214 heartbeater.cc:503] Master 127.10.225.190:44369 was elected leader, sending a full tablet report...
I20260806 23:56:05.606787 6244 raft_consensus.cc:729] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ddd53e97b904d65837e14ac8d02de41, State: Initialized, Role: FOLLOWER
I20260806 23:56:05.607003 6240 ts_tablet_manager.cc:1428] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:56:05.607230 6244 consensus_queue.cc:260] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } }
I20260806 23:56:05.606423 6243 raft_consensus.cc:374] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:05.607416 6243 raft_consensus.cc:729] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a481c0ad145846ea86ac1152f768c25c, State: Initialized, Role: FOLLOWER
I20260806 23:56:05.607925 6243 consensus_queue.cc:260] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } }
I20260806 23:56:05.608239 6244 ts_tablet_manager.cc:1428] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260806 23:56:05.608355 5783 heartbeater.cc:503] Master 127.10.225.190:44369 was elected leader, sending a full tablet report...
I20260806 23:56:05.608800 5996 heartbeater.cc:503] Master 127.10.225.190:44369 was elected leader, sending a full tablet report...
I20260806 23:56:05.609525 6243 ts_tablet_manager.cc:1428] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 23:56:05.763051 6253 raft_consensus.cc:482] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:05.763299 6253 raft_consensus.cc:504] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } }
I20260806 23:56:05.764518 6253 leader_election.cc:288] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36082c29f0d64b0da6497c3f9bb24b14 (127.10.225.131:35143), a481c0ad145846ea86ac1152f768c25c (127.10.225.130:35195)
I20260806 23:56:05.769025 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.764912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:39983 (local address 127.10.225.131:35143)
0806 23:56:05.765045 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.765050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.765301 (+ 251us) server_negotiation.cc:407] Connection header received
0806 23:56:05.765486 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.765490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.765534 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.765597 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:05.765906 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.766427 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.767824 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.768443 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.768575 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.768633 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.768814 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.768900 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:56:05.769052 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.764808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:35143 (local address 127.10.225.129:39983)
0806 23:56:05.765213 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.765225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:05.765352 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:05.765627 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:05.765630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:05.765647 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:05.765798 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.765804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.766528 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.766532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.767714 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.767726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.768579 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.768582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.768684 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.768714 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 23:56:05.768794 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 23:56:05.768870 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":76,"thread_start_us":120,"threads_started":1}
I20260806 23:56:05.769140 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.764912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:57051 (local address 127.10.225.130:35195)
0806 23:56:05.765045 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.765050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.765372 (+ 322us) server_negotiation.cc:407] Connection header received
0806 23:56:05.765489 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.765491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.765536 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.765612 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:05.765934 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.766450 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.767888 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.768496 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.768589 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.768633 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.768936 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.769012 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 23:56:05.769549 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.764808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35195 (local address 127.10.225.129:57051)
0806 23:56:05.765286 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.765298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:05.765393 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:05.765619 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:05.765621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:05.765638 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:05.765819 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.765825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.766553 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.766555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.767783 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:05.767795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.768694 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.768698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:05.768784 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.768807 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:56:05.769319 (+ 512us) client_negotiation.cc:240] Negotiation successful
0806 23:56:05.769398 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":54,"spinlock_wait_cycles":8064,"thread_start_us":96,"threads_started":1}
I20260806 23:56:05.770253 6138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c772505eb554b11a749663c39123296" candidate_uuid: "4ddd53e97b904d65837e14ac8d02de41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36082c29f0d64b0da6497c3f9bb24b14" is_pre_election: true
I20260806 23:56:05.770682 6138 raft_consensus.cc:2454] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ddd53e97b904d65837e14ac8d02de41 in term 0.
I20260806 23:56:05.771227 5619 leader_election.cc:302] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36082c29f0d64b0da6497c3f9bb24b14, 4ddd53e97b904d65837e14ac8d02de41; no voters:
I20260806 23:56:05.771549 6253 raft_consensus.cc:2781] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:05.771548 5926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c772505eb554b11a749663c39123296" candidate_uuid: "4ddd53e97b904d65837e14ac8d02de41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a481c0ad145846ea86ac1152f768c25c" is_pre_election: true
I20260806 23:56:05.771656 6253 raft_consensus.cc:482] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:05.771733 6253 raft_consensus.cc:3037] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:05.771837 5926 raft_consensus.cc:2454] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ddd53e97b904d65837e14ac8d02de41 in term 0.
W20260806 23:56:05.772190 6216 tablet.cc:2369] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:05.773101 6253 raft_consensus.cc:504] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } }
I20260806 23:56:05.773480 6253 leader_election.cc:288] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [CANDIDATE]: Term 1 election: Requested vote from peers 36082c29f0d64b0da6497c3f9bb24b14 (127.10.225.131:35143), a481c0ad145846ea86ac1152f768c25c (127.10.225.130:35195)
I20260806 23:56:05.773818 5926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c772505eb554b11a749663c39123296" candidate_uuid: "4ddd53e97b904d65837e14ac8d02de41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a481c0ad145846ea86ac1152f768c25c"
I20260806 23:56:05.773962 5926 raft_consensus.cc:3037] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:05.775245 5926 raft_consensus.cc:2454] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ddd53e97b904d65837e14ac8d02de41 in term 1.
I20260806 23:56:05.775532 6138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c772505eb554b11a749663c39123296" candidate_uuid: "4ddd53e97b904d65837e14ac8d02de41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36082c29f0d64b0da6497c3f9bb24b14"
I20260806 23:56:05.775573 5616 leader_election.cc:302] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ddd53e97b904d65837e14ac8d02de41, a481c0ad145846ea86ac1152f768c25c; no voters:
I20260806 23:56:05.775662 6138 raft_consensus.cc:3037] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:05.776935 6138 raft_consensus.cc:2454] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ddd53e97b904d65837e14ac8d02de41 in term 1.
I20260806 23:56:05.777172 6253 raft_consensus.cc:2781] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:05.777415 6253 raft_consensus.cc:686] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 1 LEADER]: Becoming Leader. State: Replica: 4ddd53e97b904d65837e14ac8d02de41, State: Running, Role: LEADER
I20260806 23:56:05.777756 6253 consensus_queue.cc:237] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } }
I20260806 23:56:05.781359 5496 catalog_manager.cc:5495] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ddd53e97b904d65837e14ac8d02de41 (127.10.225.129). New cstate: current_term: 1 leader_uuid: "4ddd53e97b904d65837e14ac8d02de41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ddd53e97b904d65837e14ac8d02de41" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 41265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 } health_report { overall_health: UNKNOWN } } }
W20260806 23:56:05.793001 5997 tablet.cc:2369] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:56:05.824172 5785 tablet.cc:2369] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:05.825532 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.821806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.10.225.129:41265)
0806 23:56:05.821928 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:05.821932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:05.821945 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:56:05.822019 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:05.822021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:05.822075 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:05.822168 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:05.822564 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.823094 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.823978 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:05.824260 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:05.824385 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:05.824702 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:05.825143 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:05.825256 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:05.825377 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 23:56:05.825419 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:56:05.831365 5926 raft_consensus.cc:1261] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [term 1 FOLLOWER]: Refusing update from remote peer 4ddd53e97b904d65837e14ac8d02de41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:05.831424 6138 raft_consensus.cc:1261] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [term 1 FOLLOWER]: Refusing update from remote peer 4ddd53e97b904d65837e14ac8d02de41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:05.832086 6253 consensus_queue.cc:1035] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36082c29f0d64b0da6497c3f9bb24b14" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:05.832273 6260 consensus_queue.cc:1035] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a481c0ad145846ea86ac1152f768c25c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:05.840207 6265 mvcc.cc:205] Tried to move back new op lower bound from 7315704077637943296 to 7315704077428592640. Current Snapshot: MvccSnapshot[applied={T|T < 7315704077637943296}]
I20260806 23:56:05.848776 5494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:05.848989 5494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:05.852432 5494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48b29520b4a244afbf7bc62b325ed61c: Sending DeleteTablet for 3 replicas of tablet 2c772505eb554b11a749663c39123296
I20260806 23:56:05.853318 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c772505eb554b11a749663c39123296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:05 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:40902
I20260806 23:56:05.853871 6280 tablet_replica.cc:330] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41: stopping tablet replica
I20260806 23:56:05.853904 6113 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c772505eb554b11a749663c39123296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:05 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:38544
I20260806 23:56:05.854235 6280 raft_consensus.cc:2229] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:56:05.854396 6282 tablet_replica.cc:330] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: stopping tablet replica
I20260806 23:56:05.854676 6280 raft_consensus.cc:2258] T 2c772505eb554b11a749663c39123296 P 4ddd53e97b904d65837e14ac8d02de41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:05.854725 6282 raft_consensus.cc:2229] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:05.854842 5897 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c772505eb554b11a749663c39123296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:05 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:35906
I20260806 23:56:05.855087 6282 raft_consensus.cc:2258] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:05.856050 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 5586
I20260806 23:56:05.859804 6282 ts_tablet_manager.cc:1905] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:05.862768 6282 ts_tablet_manager.cc:1918] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:05.862859 6282 log.cc:1196] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/2c772505eb554b11a749663c39123296
I20260806 23:56:05.863171 6282 ts_tablet_manager.cc:1939] T 2c772505eb554b11a749663c39123296 P 36082c29f0d64b0da6497c3f9bb24b14: Deleting consensus metadata
I20260806 23:56:05.864132 6283 tablet_replica.cc:330] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: stopping tablet replica
I20260806 23:56:05.864480 6283 raft_consensus.cc:2229] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:05.864619 5473 catalog_manager.cc:4841] TS 36082c29f0d64b0da6497c3f9bb24b14 (127.10.225.131:35143): tablet 2c772505eb554b11a749663c39123296 (table test-table [id=b835bb62e22b4d0fa0d877ab9bd2b86a]) successfully deleted
I20260806 23:56:05.864853 6283 raft_consensus.cc:2258] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:05.866195 6283 ts_tablet_manager.cc:1905] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 23:56:05.868109 5474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:41265 (error 108)
I20260806 23:56:05.868865 6283 ts_tablet_manager.cc:1918] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:05.868970 6283 log.cc:1196] T 2c772505eb554b11a749663c39123296 P a481c0ad145846ea86ac1152f768c25c: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/2c772505eb554b11a749663c39123296
I20260806 23:56:05.868999 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 5792
I20260806 23:56:05.869204 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.868897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:41265 (local address 127.0.0.1:40922)
0806 23:56:05.869044 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.869118 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:41265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 23:56:05.873245 5474 catalog_manager.cc:4584] TS 4ddd53e97b904d65837e14ac8d02de41 (127.10.225.129:41265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c772505eb554b11a749663c39123296: Network error: Client connection negotiation failed: client connection to 127.10.225.129:41265: connect: Connection refused (error 111)
I20260806 23:56:05.878844 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 6004
I20260806 23:56:05.879376 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:05.879050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:35195 (local address 127.0.0.1:35918)
0806 23:56:05.879219 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:05.879265 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.130:35195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 23:56:05.899075 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 5445
2026-08-06T23:56:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2580 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T23:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:05Z Disabled control of system clock
I20260806 23:56:05.938426 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:32881
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:32881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 23:56:06.055867 6288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:06.056155 6288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:06.056191 6288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:06.060376 6288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:06.060451 6288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:06.060467 6288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:06.060482 6288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:06.060497 6288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:06.065896 6288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:32881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:32881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:06.067040 6288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:06.068635 6288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:06.077258 6297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:06.077260 6300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:06.077351 6288 server_base.cc:1034] running on GCE node
W20260806 23:56:06.077260 6298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:06.078161 6288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:06.079322 6288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:06.080497 6288 hybrid_clock.cc:648] HybridClock initialized: now 1786060566080483 us; error 65 us; skew 500 ppm
I20260806 23:56:06.082370 6288 webserver.cc:460] Webserver started at http://127.10.225.190:35767/ using document root <none> and password file <none>
I20260806 23:56:06.082955 6288 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:06.083024 6288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:06.083240 6288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:06.085414 6288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "647d5db9a27342efa934df8703b1d6b1"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "978019d67ad58cad6f18b0c510be932f"
server_key_iv: "73d24b3b670150d3a9692e4ade4bfd2e"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.085968 6288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "647d5db9a27342efa934df8703b1d6b1"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "978019d67ad58cad6f18b0c510be932f"
server_key_iv: "73d24b3b670150d3a9692e4ade4bfd2e"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.089743 6288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:06.092316 6307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.093585 6288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:06.093703 6288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "647d5db9a27342efa934df8703b1d6b1"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "978019d67ad58cad6f18b0c510be932f"
server_key_iv: "73d24b3b670150d3a9692e4ade4bfd2e"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.093814 6288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:06.104404 6288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:06.105093 6288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:06.105242 6288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:06.111972 6288 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:32881
I20260806 23:56:06.112020 6383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:32881 every 8 connection(s)
I20260806 23:56:06.112906 6288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:06.114856 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 6288
I20260806 23:56:06.114984 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:06.115278 11142 external_mini_cluster.cc:1428] Setting key bdaa33fc50ffa68745329aef3a94b905
I20260806 23:56:06.116171 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:06.122201 6386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1: Bootstrap starting.
I20260806 23:56:06.124578 6386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:06.124619 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.116720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.10.225.190:32881)
0806 23:56:06.117376 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:06.117391 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:06.117426 (+ 35us) server_negotiation.cc:407] Connection header received
0806 23:56:06.118152 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:06.118170 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:06.118555 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:06.118844 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:06.119582 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.120598 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.121483 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.121954 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.122097 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.122932 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:06.122937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:06.122949 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:06.123321 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:06.123388 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:06.123409 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:06.123494 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:06.123627 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 23:56:06.123915 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":165,"threads_started":1}
I20260806 23:56:06.125478 6386 log.cc:824] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:06.127447 6386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1: No bootstrap required, opened a new log
I20260806 23:56:06.131151 6386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d5db9a27342efa934df8703b1d6b1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32881 } }
I20260806 23:56:06.131384 6386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:06.131425 6386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 647d5db9a27342efa934df8703b1d6b1, State: Initialized, Role: FOLLOWER
I20260806 23:56:06.131855 6386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d5db9a27342efa934df8703b1d6b1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32881 } }
I20260806 23:56:06.131985 6386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:06.132036 6386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:06.132124 6386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:06.133195 6386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d5db9a27342efa934df8703b1d6b1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32881 } }
I20260806 23:56:06.133522 6386 leader_election.cc:302] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 647d5db9a27342efa934df8703b1d6b1; no voters:
I20260806 23:56:06.133796 6386 leader_election.cc:288] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:06.133912 6397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:06.134106 6397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [term 1 LEADER]: Becoming Leader. State: Replica: 647d5db9a27342efa934df8703b1d6b1, State: Running, Role: LEADER
I20260806 23:56:06.134469 6397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d5db9a27342efa934df8703b1d6b1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32881 } }
I20260806 23:56:06.135015 6386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:06.136059 6398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "647d5db9a27342efa934df8703b1d6b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d5db9a27342efa934df8703b1d6b1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32881 } } }
I20260806 23:56:06.136190 6398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:06.136487 6403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 647d5db9a27342efa934df8703b1d6b1. Latest consensus state: current_term: 1 leader_uuid: "647d5db9a27342efa934df8703b1d6b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647d5db9a27342efa934df8703b1d6b1" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32881 } } }
I20260806 23:56:06.136605 6403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:06.136638 6412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:06.140661 6412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:06.146777 6412 catalog_manager.cc:1347] Generated new cluster ID: af4ac5b1020149d1abbe6fbe82f514c4
I20260806 23:56:06.146876 6412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:06.155090 6412 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:06.156219 6412 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:06.167662 6412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1: Generated new TSK 0
I20260806 23:56:06.168418 6412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:06.176482 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:32881
--builtin_ntp_servers=127.10.225.148:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 23:56:06.296319 6426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:06.296590 6426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:06.296629 6426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:06.300760 6426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:06.300824 6426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:06.300880 6426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:56:06.305773 6426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:32881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:06.306881 6426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:06.308279 6426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:06.318451 6433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:06.318454 6436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:06.318486 6426 server_base.cc:1034] running on GCE node
W20260806 23:56:06.318758 6438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:06.319168 6426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:06.320482 6426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:06.321777 6426 hybrid_clock.cc:648] HybridClock initialized: now 1786060566321741 us; error 55 us; skew 500 ppm
I20260806 23:56:06.323835 6426 webserver.cc:460] Webserver started at http://127.10.225.129:39139/ using document root <none> and password file <none>
I20260806 23:56:06.324425 6426 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:06.324491 6426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:06.324683 6426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:06.326794 6426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "40808c5bcb38433ea7f59e67a6013308"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "35fafd1bac233952deb5392d7f267ea8"
server_key_iv: "9355aa96f0039c7fe1ecbb8a55db8285"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.327324 6426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "40808c5bcb38433ea7f59e67a6013308"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "35fafd1bac233952deb5392d7f267ea8"
server_key_iv: "9355aa96f0039c7fe1ecbb8a55db8285"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.331212 6426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:56:06.334946 6446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.336457 6426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:06.336583 6426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "40808c5bcb38433ea7f59e67a6013308"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "35fafd1bac233952deb5392d7f267ea8"
server_key_iv: "9355aa96f0039c7fe1ecbb8a55db8285"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.336704 6426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:06.355835 6426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:06.356916 6426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:06.357118 6426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:06.357748 6426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:06.358741 6426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:06.358791 6426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.358855 6426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:06.358883 6426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.367259 6426 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:40749
I20260806 23:56:06.367321 6617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:40749 every 8 connection(s)
I20260806 23:56:06.368180 6426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:56:06.374768 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 6426
I20260806 23:56:06.374914 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:06.375226 11142 external_mini_cluster.cc:1428] Setting key 1fd0d73186091378f49f1307550c5482
I20260806 23:56:06.377187 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:32881
--builtin_ntp_servers=127.10.225.148:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 23:56:06.377878 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.370010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:44467 (local address 127.10.225.190:32881)
0806 23:56:06.370155 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:06.370159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:06.371179 (+ 1020us) server_negotiation.cc:407] Connection header received
0806 23:56:06.372611 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:06.372615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:06.372679 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:06.372750 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:06.373622 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.374207 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.375831 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.376189 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.376305 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.377397 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:06.377400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:06.377403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:06.377416 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:06.377435 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:06.377443 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:06.377476 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:06.377662 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 23:56:06.377734 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:56:06.378719 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.370375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:32881 (local address 127.10.225.129:44467)
0806 23:56:06.371033 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:06.371073 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:06.372412 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:06.372925 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:06.372932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:06.373009 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:06.373485 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.373498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.374357 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.374361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.375688 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.375701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.376362 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.376367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.376545 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.377116 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:56:06.377118 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:06.377158 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:56:06.377223 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:06.377526 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:06.377533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:06.377537 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 23:56:06.377634 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 23:56:06.377926 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260806 23:56:06.379962 6618 heartbeater.cc:347] Connected to a master server at 127.10.225.190:32881
I20260806 23:56:06.380190 6618 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:06.380704 6618 heartbeater.cc:511] Master 127.10.225.190:32881 requested a full tablet report, sending...
I20260806 23:56:06.382334 6332 ts_manager.cc:194] Registered new tserver with Master: 40808c5bcb38433ea7f59e67a6013308 (127.10.225.129:40749)
I20260806 23:56:06.383960 6332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.129:44467
W20260806 23:56:06.491042 6623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:06.491310 6623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:06.491356 6623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:06.495299 6623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:06.495366 6623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:06.495434 6623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:56:06.501199 6623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:32881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:06.502313 6623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:06.503702 6623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:06.512992 6630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:06.513010 6632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:06.512991 6629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:06.513391 6623 server_base.cc:1034] running on GCE node
I20260806 23:56:06.513748 6623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:06.514359 6623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:06.515519 6623 hybrid_clock.cc:648] HybridClock initialized: now 1786060566515496 us; error 36 us; skew 500 ppm
I20260806 23:56:06.517318 6623 webserver.cc:460] Webserver started at http://127.10.225.130:36065/ using document root <none> and password file <none>
I20260806 23:56:06.517872 6623 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:06.517932 6623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:06.518131 6623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:06.520234 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "4c5d22a1d8a740b5a554fe82d274f75c"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "6665a091e0953c07ac2e3116569ee9ae"
server_key_iv: "db85d6c59dca008928ccbe962f7cb14e"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.520778 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "4c5d22a1d8a740b5a554fe82d274f75c"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "6665a091e0953c07ac2e3116569ee9ae"
server_key_iv: "db85d6c59dca008928ccbe962f7cb14e"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.524345 6623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:56:06.526736 6641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.527936 6623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 23:56:06.528048 6623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "4c5d22a1d8a740b5a554fe82d274f75c"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "6665a091e0953c07ac2e3116569ee9ae"
server_key_iv: "db85d6c59dca008928ccbe962f7cb14e"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.528151 6623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:06.537703 6623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:06.538362 6623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:06.538527 6623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:06.539104 6623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:06.540131 6623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:06.540177 6623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.540222 6623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:06.540251 6623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.549109 6623 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:37183
I20260806 23:56:06.549144 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:37183 every 8 connection(s)
I20260806 23:56:06.550063 6623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:56:06.554541 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 6623
I20260806 23:56:06.554665 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:06.554965 11142 external_mini_cluster.cc:1428] Setting key 4c4f8abbcabf162d86041b3c7cb4c384
I20260806 23:56:06.557108 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:32881
--builtin_ntp_servers=127.10.225.148:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 23:56:06.559887 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.551926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:39921 (local address 127.10.225.190:32881)
0806 23:56:06.552075 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:06.552080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:06.553142 (+ 1062us) server_negotiation.cc:407] Connection header received
0806 23:56:06.554541 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:06.554544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:06.554605 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:06.554675 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:06.555664 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.556420 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.557970 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.558183 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.558258 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.559408 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:06.559419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:06.559421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:06.559433 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:06.559453 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:06.559460 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:06.559517 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:06.559655 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 23:56:06.559744 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:56:06.560791 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.552346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:32881 (local address 127.10.225.130:39921)
0806 23:56:06.553008 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:06.553046 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:06.554327 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:06.554892 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:06.554899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:06.554987 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:06.555515 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.555533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.556556 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.556560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.557836 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.557848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.558321 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.558324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.558513 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.559163 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:56:06.559164 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:06.559201 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:56:06.559267 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:06.559528 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:06.559537 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:06.559543 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:56:06.559649 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 23:56:06.559968 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260806 23:56:06.561887 6815 heartbeater.cc:347] Connected to a master server at 127.10.225.190:32881
I20260806 23:56:06.562120 6815 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:06.562626 6815 heartbeater.cc:511] Master 127.10.225.190:32881 requested a full tablet report, sending...
I20260806 23:56:06.563702 6332 ts_manager.cc:194] Registered new tserver with Master: 4c5d22a1d8a740b5a554fe82d274f75c (127.10.225.130:37183)
I20260806 23:56:06.564471 6332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.130:39921
W20260806 23:56:06.669394 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:06.669673 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:06.669713 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:06.673708 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:06.673766 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:06.673823 6819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:56:06.678692 6819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:32881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:06.679802 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:06.681154 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:06.690649 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:06.690649 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:06.690670 6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:06.691025 6819 server_base.cc:1034] running on GCE node
I20260806 23:56:06.691471 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:06.692140 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:06.693284 6819 hybrid_clock.cc:648] HybridClock initialized: now 1786060566693261 us; error 36 us; skew 500 ppm
I20260806 23:56:06.695173 6819 webserver.cc:460] Webserver started at http://127.10.225.131:36041/ using document root <none> and password file <none>
I20260806 23:56:06.695768 6819 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:06.695844 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:06.696049 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:06.698194 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "bc7e404cc4ab41bc94839cf511845601"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "7a1028e411058e23810c6376b03cc758"
server_key_iv: "1a3d9a888f23d206f347ce04e03aa2b1"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.698936 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "bc7e404cc4ab41bc94839cf511845601"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "7a1028e411058e23810c6376b03cc758"
server_key_iv: "1a3d9a888f23d206f347ce04e03aa2b1"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.702930 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:56:06.705467 6839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.706763 6819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:06.706885 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "bc7e404cc4ab41bc94839cf511845601"
format_stamp: "Formatted at 2026-08-06 23:56:06 on dist-test-slave-nj21"
server_key: "7a1028e411058e23810c6376b03cc758"
server_key_iv: "1a3d9a888f23d206f347ce04e03aa2b1"
server_key_version: "encryptionkey@0"
I20260806 23:56:06.706986 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:06.718370 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:06.719101 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:06.719275 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:06.719908 6819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:06.720883 6819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:06.720932 6819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.720985 6819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:06.721010 6819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:06.730261 6819 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:41727
I20260806 23:56:06.730346 7016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:41727 every 8 connection(s)
I20260806 23:56:06.731225 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:56:06.734405 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 6819
I20260806 23:56:06.734536 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:56:06.734836 11142 external_mini_cluster.cc:1428] Setting key 503a02ce3b2fa409ab26495c9a16ed72
I20260806 23:56:06.740952 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.733332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:42661 (local address 127.10.225.190:32881)
0806 23:56:06.733477 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:06.733481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:06.734555 (+ 1074us) server_negotiation.cc:407] Connection header received
0806 23:56:06.736318 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:06.736321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:06.736374 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:06.736474 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:06.737451 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.738026 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.739101 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.739319 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.739397 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.740560 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:06.740563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:06.740565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:06.740577 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:06.740596 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:06.740607 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:06.740688 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:06.740787 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 23:56:06.740837 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 23:56:06.741822 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.733649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:32881 (local address 127.10.225.131:42661)
0806 23:56:06.734371 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:06.734425 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:06.736114 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:06.736632 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:06.736640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:06.736753 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:06.737328 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.737342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.738124 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.738126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.738993 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.739002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.739478 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.739481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.739637 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.740310 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:56:06.740312 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:06.740352 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:56:06.740417 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:06.740686 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:06.740691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:06.740694 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 23:56:06.740790 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 23:56:06.741015 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20260806 23:56:06.742935 7017 heartbeater.cc:347] Connected to a master server at 127.10.225.190:32881
I20260806 23:56:06.743157 7017 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:06.743656 7017 heartbeater.cc:511] Master 127.10.225.190:32881 requested a full tablet report, sending...
I20260806 23:56:06.744691 6332 ts_manager.cc:194] Registered new tserver with Master: bc7e404cc4ab41bc94839cf511845601 (127.10.225.131:41727)
I20260806 23:56:06.745291 6332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.131:42661
I20260806 23:56:06.748904 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:56:06.754380 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.750893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.10.225.190:32881)
0806 23:56:06.751000 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:06.751004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:06.751099 (+ 95us) server_negotiation.cc:407] Connection header received
0806 23:56:06.751192 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:06.751194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:06.751231 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:06.751299 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:06.751618 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.752149 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.752940 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.753161 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.753259 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.753994 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:06.753997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:06.753999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:06.754012 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:06.754031 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:06.754039 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:06.754138 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:06.754230 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 23:56:06.754265 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 23:56:06.757195 6332 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:56:06.760299 6332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:06.774394 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.768374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:41727 (local address 127.0.0.1:53904)
0806 23:56:06.768720 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:06.768768 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:06.768897 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:06.770699 (+ 1802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:06.770702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:06.770719 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:06.770796 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.770802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.771787 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.771791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.772941 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.772951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.773842 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.773845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.773957 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.774000 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 23:56:06.774118 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 23:56:06.774240 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":119,"thread_start_us":76,"threads_started":1}
I20260806 23:56:06.774435 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.768477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53904 (local address 127.10.225.131:41727)
0806 23:56:06.770374 (+ 1897us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:06.770386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:06.770404 (+ 18us) server_negotiation.cc:407] Connection header received
0806 23:56:06.770463 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:06.770466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:06.770587 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:06.770702 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:06.770898 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.771617 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.773058 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.773714 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.773835 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.774031 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:06.774159 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 23:56:06.774265 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1777,"thread_start_us":169,"threads_started":1}
I20260806 23:56:06.774928 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.768780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:37183 (local address 127.0.0.1:56150)
0806 23:56:06.769373 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:06.769386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:06.769490 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:06.770061 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:06.770066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:06.770103 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:06.770254 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.770261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.771723 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.771727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.773306 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.773318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.774511 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.774515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.774601 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.774649 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 23:56:06.774718 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 23:56:06.774789 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":164,"spinlock_wait_cycles":3840,"thread_start_us":102,"threads_started":1}
I20260806 23:56:06.775014 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.769198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.10.225.130:37183)
0806 23:56:06.769608 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:06.769621 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:06.769644 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:56:06.769725 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:06.769731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:06.769923 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:06.770059 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:06.770368 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.771553 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.773439 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.774160 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.774274 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.774440 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:06.774749 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 23:56:06.774842 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":95,"threads_started":1}
I20260806 23:56:06.775244 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.769064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:40749 (local address 127.0.0.1:42298)
0806 23:56:06.769651 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:06.769663 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:06.769730 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:06.770068 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:06.770070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:06.770101 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:06.770258 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.770266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.771749 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.771751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.773306 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:06.773318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.774605 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.774608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:06.774751 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.774779 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 23:56:06.774841 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 23:56:06.774917 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":266,"spinlock_wait_cycles":3840,"thread_start_us":84,"threads_started":1}
I20260806 23:56:06.777737 6917 tablet_service.cc:1467] Processing CreateTablet for tablet 2a17427f78134df98cfc026b4745d2e8 (DEFAULT_TABLE table=test-table [id=758adb5baee5458b8e23328e52fbe2b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:06.778483 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:06.769255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.10.225.129:40749)
0806 23:56:06.769711 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:06.769720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:06.769739 (+ 19us) server_negotiation.cc:407] Connection header received
0806 23:56:06.769806 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:06.769809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:06.769945 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:06.770071 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:06.770405 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.771568 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.773455 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:06.774047 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:06.774156 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:06.774306 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:06.774992 (+ 686us) server_negotiation.cc:299] Negotiation successful
0806 23:56:06.778319 (+ 3327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":156,"threads_started":1}
I20260806 23:56:06.778748 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a17427f78134df98cfc026b4745d2e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:06.780222 6708 tablet_service.cc:1467] Processing CreateTablet for tablet 2a17427f78134df98cfc026b4745d2e8 (DEFAULT_TABLE table=test-table [id=758adb5baee5458b8e23328e52fbe2b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:06.781190 6708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a17427f78134df98cfc026b4745d2e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:06.781055 6516 tablet_service.cc:1467] Processing CreateTablet for tablet 2a17427f78134df98cfc026b4745d2e8 (DEFAULT_TABLE table=test-table [id=758adb5baee5458b8e23328e52fbe2b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:06.781992 6516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a17427f78134df98cfc026b4745d2e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:06.786738 7044 tablet_bootstrap.cc:492] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: Bootstrap starting.
I20260806 23:56:06.787825 7046 tablet_bootstrap.cc:492] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: Bootstrap starting.
I20260806 23:56:06.788520 7047 tablet_bootstrap.cc:492] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308: Bootstrap starting.
I20260806 23:56:06.789247 7044 tablet_bootstrap.cc:654] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:06.789877 7046 tablet_bootstrap.cc:654] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:06.790179 7044 log.cc:824] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:06.790293 7047 tablet_bootstrap.cc:654] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:06.790657 7046 log.cc:824] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:06.791051 7047 log.cc:824] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:06.792582 7044 tablet_bootstrap.cc:492] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: No bootstrap required, opened a new log
I20260806 23:56:06.792582 7046 tablet_bootstrap.cc:492] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: No bootstrap required, opened a new log
I20260806 23:56:06.792781 7044 ts_tablet_manager.cc:1397] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260806 23:56:06.792781 7046 ts_tablet_manager.cc:1397] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 23:56:06.792814 7047 tablet_bootstrap.cc:492] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308: No bootstrap required, opened a new log
I20260806 23:56:06.792981 7047 ts_tablet_manager.cc:1397] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 23:56:06.796154 7047 raft_consensus.cc:348] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } }
I20260806 23:56:06.796389 7047 raft_consensus.cc:374] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:06.796463 7047 raft_consensus.cc:729] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40808c5bcb38433ea7f59e67a6013308, State: Initialized, Role: FOLLOWER
I20260806 23:56:06.796931 7047 consensus_queue.cc:260] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } }
I20260806 23:56:06.796988 7046 raft_consensus.cc:348] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } }
I20260806 23:56:06.796988 7044 raft_consensus.cc:348] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } }
I20260806 23:56:06.797230 7044 raft_consensus.cc:374] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:06.797300 7046 raft_consensus.cc:374] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:06.797333 7044 raft_consensus.cc:729] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7e404cc4ab41bc94839cf511845601, State: Initialized, Role: FOLLOWER
I20260806 23:56:06.797410 7046 raft_consensus.cc:729] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5d22a1d8a740b5a554fe82d274f75c, State: Initialized, Role: FOLLOWER
I20260806 23:56:06.797595 6618 heartbeater.cc:503] Master 127.10.225.190:32881 was elected leader, sending a full tablet report...
I20260806 23:56:06.797873 7046 consensus_queue.cc:260] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } }
I20260806 23:56:06.797973 7047 ts_tablet_manager.cc:1428] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 23:56:06.798759 6815 heartbeater.cc:503] Master 127.10.225.190:32881 was elected leader, sending a full tablet report...
I20260806 23:56:06.799094 7044 consensus_queue.cc:260] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } }
I20260806 23:56:06.799733 7017 heartbeater.cc:503] Master 127.10.225.190:32881 was elected leader, sending a full tablet report...
I20260806 23:56:06.800093 7044 ts_tablet_manager.cc:1428] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 23:56:06.800436 7046 ts_tablet_manager.cc:1428] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260806 23:56:06.801131 6816 tablet.cc:2369] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:56:06.869340 6621 tablet.cc:2369] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 23:56:06.982331 7018 tablet.cc:2369] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:07.092049 7055 raft_consensus.cc:482] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:07.092289 7055 raft_consensus.cc:504] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } }
I20260806 23:56:07.093415 7055 leader_election.cc:288] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc7e404cc4ab41bc94839cf511845601 (127.10.225.131:41727), 4c5d22a1d8a740b5a554fe82d274f75c (127.10.225.130:37183)
I20260806 23:56:07.097412 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:07.093913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:42711 (local address 127.10.225.131:41727)
0806 23:56:07.094025 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:07.094029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:07.094116 (+ 87us) server_negotiation.cc:407] Connection header received
0806 23:56:07.094272 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:07.094276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:07.094320 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:07.094382 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:07.094639 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.095148 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.096408 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.096873 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.096978 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:07.097048 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:07.097198 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 23:56:07.097272 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:56:07.097426 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:07.093805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:44365 (local address 127.10.225.130:37183)
0806 23:56:07.094053 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:07.094058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:07.094175 (+ 117us) server_negotiation.cc:407] Connection header received
0806 23:56:07.094310 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:07.094314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:07.094357 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:07.094421 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:07.094669 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.095159 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.096407 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.096852 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.096957 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:07.097012 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:07.097228 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 23:56:07.097297 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260806 23:56:07.097448 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:07.093707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:41727 (local address 127.10.225.129:42711)
0806 23:56:07.094025 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:07.094040 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:07.094168 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:07.094401 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:07.094404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:07.094419 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:07.094542 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:07.094547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.095247 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.095251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:07.096291 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:07.096301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.096978 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.096982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:07.097074 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:07.097113 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 23:56:07.097175 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 23:56:07.097282 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":100,"thread_start_us":95,"threads_started":1}
I20260806 23:56:07.097452 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:07.093707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:37183 (local address 127.10.225.129:44365)
0806 23:56:07.094096 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:07.094108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:07.094208 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:07.094445 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:07.094448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:07.094464 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:07.094578 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:07.094583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.095263 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.095266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:07.096286 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:07.096302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.096966 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.096969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:07.097064 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:07.097119 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 23:56:07.097184 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 23:56:07.097280 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":99,"thread_start_us":78,"threads_started":1}
I20260806 23:56:07.098168 6740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a17427f78134df98cfc026b4745d2e8" candidate_uuid: "40808c5bcb38433ea7f59e67a6013308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" is_pre_election: true
I20260806 23:56:07.098490 6953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a17427f78134df98cfc026b4745d2e8" candidate_uuid: "40808c5bcb38433ea7f59e67a6013308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc7e404cc4ab41bc94839cf511845601" is_pre_election: true
I20260806 23:56:07.098537 6740 raft_consensus.cc:2454] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40808c5bcb38433ea7f59e67a6013308 in term 0.
I20260806 23:56:07.098752 6953 raft_consensus.cc:2454] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40808c5bcb38433ea7f59e67a6013308 in term 0.
I20260806 23:56:07.099011 6450 leader_election.cc:302] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40808c5bcb38433ea7f59e67a6013308, 4c5d22a1d8a740b5a554fe82d274f75c; no voters:
I20260806 23:56:07.099300 7055 raft_consensus.cc:2781] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:07.099386 7055 raft_consensus.cc:482] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:07.099421 7055 raft_consensus.cc:3037] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:07.100492 7055 raft_consensus.cc:504] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } }
I20260806 23:56:07.100845 7055 leader_election.cc:288] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [CANDIDATE]: Term 1 election: Requested vote from peers bc7e404cc4ab41bc94839cf511845601 (127.10.225.131:41727), 4c5d22a1d8a740b5a554fe82d274f75c (127.10.225.130:37183)
I20260806 23:56:07.101202 6953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a17427f78134df98cfc026b4745d2e8" candidate_uuid: "40808c5bcb38433ea7f59e67a6013308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc7e404cc4ab41bc94839cf511845601"
I20260806 23:56:07.101204 6740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a17427f78134df98cfc026b4745d2e8" candidate_uuid: "40808c5bcb38433ea7f59e67a6013308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5d22a1d8a740b5a554fe82d274f75c"
I20260806 23:56:07.101337 6953 raft_consensus.cc:3037] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:07.101336 6740 raft_consensus.cc:3037] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:07.102602 6953 raft_consensus.cc:2454] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40808c5bcb38433ea7f59e67a6013308 in term 1.
I20260806 23:56:07.102602 6740 raft_consensus.cc:2454] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40808c5bcb38433ea7f59e67a6013308 in term 1.
I20260806 23:56:07.102922 6449 leader_election.cc:302] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40808c5bcb38433ea7f59e67a6013308, bc7e404cc4ab41bc94839cf511845601; no voters:
I20260806 23:56:07.103083 7055 raft_consensus.cc:2781] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:07.103310 7055 raft_consensus.cc:686] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 1 LEADER]: Becoming Leader. State: Replica: 40808c5bcb38433ea7f59e67a6013308, State: Running, Role: LEADER
I20260806 23:56:07.103631 7055 consensus_queue.cc:237] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } }
I20260806 23:56:07.107066 6331 catalog_manager.cc:5495] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40808c5bcb38433ea7f59e67a6013308 (127.10.225.129). New cstate: current_term: 1 leader_uuid: "40808c5bcb38433ea7f59e67a6013308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40808c5bcb38433ea7f59e67a6013308" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 40749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 } health_report { overall_health: UNKNOWN } } }
I20260806 23:56:07.141347 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:07.137808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.10.225.129:40749)
0806 23:56:07.137934 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:07.137938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:07.137951 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:56:07.138025 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:07.138027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:07.138083 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:07.138167 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:07.138414 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.138918 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.139822 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:07.140047 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:07.140133 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:07.140490 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:07.140932 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:07.141068 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:07.141161 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 23:56:07.141195 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:56:07.147465 6953 raft_consensus.cc:1261] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [term 1 FOLLOWER]: Refusing update from remote peer 40808c5bcb38433ea7f59e67a6013308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:07.147465 6740 raft_consensus.cc:1261] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [term 1 FOLLOWER]: Refusing update from remote peer 40808c5bcb38433ea7f59e67a6013308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:07.148293 7060 consensus_queue.cc:1035] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc7e404cc4ab41bc94839cf511845601" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:07.148588 7055 consensus_queue.cc:1035] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5d22a1d8a740b5a554fe82d274f75c" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 37183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:07.157701 7070 mvcc.cc:205] Tried to move back new op lower bound from 7315704083027886080 to 7315704082859409408. Current Snapshot: MvccSnapshot[applied={T|T < 7315704083027886080}]
I20260806 23:56:07.158511 7069 mvcc.cc:205] Tried to move back new op lower bound from 7315704083027886080 to 7315704082859409408. Current Snapshot: MvccSnapshot[applied={T|T < 7315704083027886080}]
I20260806 23:56:07.158811 7067 mvcc.cc:205] Tried to move back new op lower bound from 7315704083027886080 to 7315704082859409408. Current Snapshot: MvccSnapshot[applied={T|T < 7315704083027886080 or (T in {7315704083027886080})}]
I20260806 23:56:07.164768 6331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:07.164929 6331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:07.167287 6331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 647d5db9a27342efa934df8703b1d6b1: Sending DeleteTablet for 3 replicas of tablet 2a17427f78134df98cfc026b4745d2e8
I20260806 23:56:07.168223 6917 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a17427f78134df98cfc026b4745d2e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:07 UTC) from {username='slave'} at 127.0.0.1:53904
I20260806 23:56:07.168448 6516 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a17427f78134df98cfc026b4745d2e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:07 UTC) from {username='slave'} at 127.0.0.1:42298
I20260806 23:56:07.168444 6708 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a17427f78134df98cfc026b4745d2e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:07 UTC) from {username='slave'} at 127.0.0.1:56150
I20260806 23:56:07.168934 7081 tablet_replica.cc:330] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308: stopping tablet replica
I20260806 23:56:07.169116 7080 tablet_replica.cc:330] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: stopping tablet replica
I20260806 23:56:07.169353 7081 raft_consensus.cc:2229] T 2a17427f78134df98cfc026b4745d2e8 P 40808c5bcb38433ea7f59e67a6013308 [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:56:07.169430 7080 raft_consensus.cc:2229] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:07.169737 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 6426
I20260806 23:56:07.169844 7080 raft_consensus.cc:2258] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:07.172649 7082 tablet_replica.cc:330] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: stopping tablet replica
I20260806 23:56:07.172959 7082 raft_consensus.cc:2229] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:07.173259 7082 raft_consensus.cc:2258] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:07.175244 7082 ts_tablet_manager.cc:1905] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:07.176288 7080 ts_tablet_manager.cc:1905] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:07.178151 7082 ts_tablet_manager.cc:1918] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:07.178257 7082 log.cc:1196] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/2a17427f78134df98cfc026b4745d2e8
I20260806 23:56:07.178588 7082 ts_tablet_manager.cc:1939] T 2a17427f78134df98cfc026b4745d2e8 P 4c5d22a1d8a740b5a554fe82d274f75c: Deleting consensus metadata
I20260806 23:56:07.179044 7080 ts_tablet_manager.cc:1918] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:07.179142 7080 log.cc:1196] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/2a17427f78134df98cfc026b4745d2e8
I20260806 23:56:07.179483 7080 ts_tablet_manager.cc:1939] T 2a17427f78134df98cfc026b4745d2e8 P bc7e404cc4ab41bc94839cf511845601: Deleting consensus metadata
I20260806 23:56:07.180055 6316 catalog_manager.cc:4841] TS 4c5d22a1d8a740b5a554fe82d274f75c (127.10.225.130:37183): tablet 2a17427f78134df98cfc026b4745d2e8 (table test-table [id=758adb5baee5458b8e23328e52fbe2b4]) successfully deleted
I20260806 23:56:07.180627 6313 catalog_manager.cc:4841] TS bc7e404cc4ab41bc94839cf511845601 (127.10.225.131:41727): tablet 2a17427f78134df98cfc026b4745d2e8 (table test-table [id=758adb5baee5458b8e23328e52fbe2b4]) successfully deleted
W20260806 23:56:07.181211 6313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:40749 (error 108)
I20260806 23:56:07.181456 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 6623
I20260806 23:56:07.184451 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:07.184129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:40749 (local address 127.0.0.1:42310)
0806 23:56:07.184276 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:07.184359 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 23:56:07.184752 6313 catalog_manager.cc:4584] TS 40808c5bcb38433ea7f59e67a6013308 (127.10.225.129:40749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a17427f78134df98cfc026b4745d2e8: Network error: Client connection negotiation failed: client connection to 127.10.225.129:40749: connect: Connection refused (error 111)
I20260806 23:56:07.192355 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 6819
I20260806 23:56:07.208501 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 6288
2026-08-06T23:56:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1307 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:07 dist-test-slave-nj21 krb5kdc[7088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:07 dist-test-slave-nj21 krb5kdc[7088](info): set up 2 sockets
Aug 06 23:56:07 dist-test-slave-nj21 krb5kdc[7088](info): commencing operation
krb5kdc: starting...
W20260806 23:56:08.348660 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:08 dist-test-slave-nj21 krb5kdc[7088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:08.476923 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:45521
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:45521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:08.593394 7105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:08.593690 7105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:08.593751 7105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:08.597788 7105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:08.597862 7105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:08.597890 7105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:08.597911 7105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:08.597934 7105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:08.603693 7105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:45521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:45521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:08.604856 7105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:08.605907 7105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:56:08.614679 7105 server_base.cc:1034] running on GCE node
W20260806 23:56:08.614758 7120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:08.614642 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:08.614634 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:08.615532 7105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:08.616906 7105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:08.618098 7105 hybrid_clock.cc:648] HybridClock initialized: now 1786060568618079 us; error 33 us; skew 500 ppm
Aug 06 23:56:08 dist-test-slave-nj21 krb5kdc[7088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060568, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:08.621486 7105 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:08.622462 7105 webserver.cc:460] Webserver started at http://127.10.225.190:33775/ using document root <none> and password file <none>
I20260806 23:56:08.623059 7105 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:08.623126 7105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:08.623330 7105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:08.625500 7105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "c92bc644ef9c4038b678a11766596bb0"
format_stamp: "Formatted at 2026-08-06 23:56:08 on dist-test-slave-nj21"
server_key: "ad492835ca18c016a238be0df7c84d71"
server_key_iv: "d8f4889b46bd600f1d9002e89df1552a"
server_key_version: "encryptionkey@0"
I20260806 23:56:08.626075 7105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "c92bc644ef9c4038b678a11766596bb0"
format_stamp: "Formatted at 2026-08-06 23:56:08 on dist-test-slave-nj21"
server_key: "ad492835ca18c016a238be0df7c84d71"
server_key_iv: "d8f4889b46bd600f1d9002e89df1552a"
server_key_version: "encryptionkey@0"
I20260806 23:56:08.630436 7105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:08.633167 7129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:08.635178 7105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:56:08.635371 7105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "c92bc644ef9c4038b678a11766596bb0"
format_stamp: "Formatted at 2026-08-06 23:56:08 on dist-test-slave-nj21"
server_key: "ad492835ca18c016a238be0df7c84d71"
server_key_iv: "d8f4889b46bd600f1d9002e89df1552a"
server_key_version: "encryptionkey@0"
I20260806 23:56:08.635519 7105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:08.652359 7105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:08.655602 7105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:08.655809 7105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:08.662817 7105 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:45521
I20260806 23:56:08.662848 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:45521 every 8 connection(s)
I20260806 23:56:08.663808 7105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:08.664757 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 7105
I20260806 23:56:08.664891 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:08.665175 11142 external_mini_cluster.cc:1428] Setting key 8763021fe032ea3c88129427dde2675b
I20260806 23:56:08.667055 7209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:08.674203 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0: Bootstrap starting.
Aug 06 23:56:08 dist-test-slave-nj21 krb5kdc[7088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:08.676990 7209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:08.677721 7209 log.cc:824] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:08.680212 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0: No bootstrap required, opened a new log
I20260806 23:56:08.681224 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:08.666619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.10.225.190:45521)
0806 23:56:08.667057 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:08.667077 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:08.667122 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:56:08.668022 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:08.668046 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:08.668455 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:08.668815 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:08.669637 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:08.671157 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:08.672116 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:08.672793 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:08.672987 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:08.675803 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:08.675808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:08.675824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:08.675857 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:08.677661 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:08.678166 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:08.678173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:08.678180 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:08.678357 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:08.678676 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:08.678679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:08.678680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:08.679103 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:08.679252 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:08.679596 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 23:56:08.679956 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":135,"threads_started":1}
I20260806 23:56:08.684209 7209 raft_consensus.cc:348] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92bc644ef9c4038b678a11766596bb0" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 45521 } }
I20260806 23:56:08.684468 7209 raft_consensus.cc:374] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:08.684505 7209 raft_consensus.cc:729] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92bc644ef9c4038b678a11766596bb0, State: Initialized, Role: FOLLOWER
I20260806 23:56:08.684960 7209 consensus_queue.cc:260] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92bc644ef9c4038b678a11766596bb0" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 45521 } }
I20260806 23:56:08.685091 7209 raft_consensus.cc:388] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:08.685149 7209 raft_consensus.cc:482] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:08.685252 7209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:08.686312 7209 raft_consensus.cc:504] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92bc644ef9c4038b678a11766596bb0" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 45521 } }
I20260806 23:56:08.686628 7209 leader_election.cc:302] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c92bc644ef9c4038b678a11766596bb0; no voters:
I20260806 23:56:08.686928 7209 leader_election.cc:288] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:08.687104 7216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:08.687476 7216 raft_consensus.cc:686] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [term 1 LEADER]: Becoming Leader. State: Replica: c92bc644ef9c4038b678a11766596bb0, State: Running, Role: LEADER
I20260806 23:56:08.687906 7216 consensus_queue.cc:237] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92bc644ef9c4038b678a11766596bb0" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 45521 } }
I20260806 23:56:08.688226 7209 sys_catalog.cc:564] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:08.690716 7220 sys_catalog.cc:455] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c92bc644ef9c4038b678a11766596bb0. Latest consensus state: current_term: 1 leader_uuid: "c92bc644ef9c4038b678a11766596bb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92bc644ef9c4038b678a11766596bb0" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 45521 } } }
I20260806 23:56:08.690791 7220 sys_catalog.cc:458] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:08.690538 7217 sys_catalog.cc:455] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c92bc644ef9c4038b678a11766596bb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92bc644ef9c4038b678a11766596bb0" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 45521 } } }
I20260806 23:56:08.691207 7217 sys_catalog.cc:458] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0 [sys.catalog]: This master's current role is: LEADER
W20260806 23:56:08.693080 7240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:08.693149 7240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:08.693228 7241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:08.693964 7241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:08.699301 7241 catalog_manager.cc:1347] Generated new cluster ID: f4182042ee7741b2931b6fb9246f8970
I20260806 23:56:08.699393 7241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:08.726485 7241 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:08.727890 7241 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:08.733506 7241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0: Generated new TSK 0
I20260806 23:56:08.734288 7241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.225.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
Entry for principal kudu/127.10.225.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab.
I20260806 23:56:08.777006 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:45521
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:08.894464 7251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:08.894840 7251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:08.894912 7251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:08.898979 7251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:08.899051 7251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:08.899134 7251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:56:08.904597 7251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.129.keytab
--principal=kudu/127.10.225.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:45521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:08.905717 7251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:08.906746 7251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:08.916253 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:08.916333 7251 server_base.cc:1034] running on GCE node
W20260806 23:56:08.916234 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:08.916227 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:08.916953 7251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:08.917550 7251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:08.918706 7251 hybrid_clock.cc:648] HybridClock initialized: now 1786060568918691 us; error 26 us; skew 500 ppm
Aug 06 23:56:08 dist-test-slave-nj21 krb5kdc[7088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060568, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:08.921540 7251 init.cc:377] Logged in from keytab as kudu/127.10.225.129@KRBTEST.COM (short username kudu)
I20260806 23:56:08.922436 7251 webserver.cc:460] Webserver started at http://127.10.225.129:36909/ using document root <none> and password file <none>
I20260806 23:56:08.922976 7251 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:08.923035 7251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:08.923264 7251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:08.925352 7251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "a23cdf8da0c144b192810ae3be6755c9"
format_stamp: "Formatted at 2026-08-06 23:56:08 on dist-test-slave-nj21"
server_key: "d2371c5e028e377e8a51df74a2b49399"
server_key_iv: "59f003e1b9be8e1efdef0a30d659a0e9"
server_key_version: "encryptionkey@0"
I20260806 23:56:08.925904 7251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "a23cdf8da0c144b192810ae3be6755c9"
format_stamp: "Formatted at 2026-08-06 23:56:08 on dist-test-slave-nj21"
server_key: "d2371c5e028e377e8a51df74a2b49399"
server_key_iv: "59f003e1b9be8e1efdef0a30d659a0e9"
server_key_version: "encryptionkey@0"
I20260806 23:56:08.929629 7251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260806 23:56:08.932826 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:08.934523 7251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:56:08.934640 7251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "a23cdf8da0c144b192810ae3be6755c9"
format_stamp: "Formatted at 2026-08-06 23:56:08 on dist-test-slave-nj21"
server_key: "d2371c5e028e377e8a51df74a2b49399"
server_key_iv: "59f003e1b9be8e1efdef0a30d659a0e9"
server_key_version: "encryptionkey@0"
I20260806 23:56:08.934789 7251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:08.963909 7251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:08.967106 7251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:08.967273 7251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:08.967856 7251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:08.968653 7251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:08.968699 7251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:08.968739 7251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:08.968760 7251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:08.977458 7251 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:45105
I20260806 23:56:08.977665 7440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:45105 every 8 connection(s)
I20260806 23:56:08.978410 7251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:56:08.985262 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 7251
I20260806 23:56:08.985392 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:08.985692 11142 external_mini_cluster.cc:1428] Setting key f81d367428a41d54a07bf55e889eb9b3
Aug 06 23:56:08 dist-test-slave-nj21 krb5kdc[7088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060568, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.129@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:08.994606 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:08.980337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:46315 (local address 127.10.225.190:45521)
0806 23:56:08.980486 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:08.980489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:08.981291 (+ 802us) server_negotiation.cc:407] Connection header received
0806 23:56:08.982646 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:08.982650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:08.982702 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:08.982781 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:08.984020 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:08.984688 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:08.986383 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:08.986685 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:08.986767 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:08.989800 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:08.989805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:08.989808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:08.989840 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:08.991903 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:08.992781 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:08.992784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:08.992786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:08.992884 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:08.993319 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:08.993322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:08.993324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:08.993589 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:08.993699 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:08.994260 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 23:56:08.994420 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 23:56:08.995433 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:08.980671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:45521 (local address 127.10.225.129:46315)
0806 23:56:08.981140 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:08.981182 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:08.982420 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:08.982983 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:08.982989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:08.983352 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:08.983883 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:08.983897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:08.984838 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:08.984841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:08.986238 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:08.986255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:08.986854 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:08.986859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:08.987093 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:08.987820 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:08.987837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:08.989547 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:08.992039 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:08.992045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:08.992060 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:08.992593 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:08.993015 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:08.993018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:08.993020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:08.993185 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:08.993711 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:08.993717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:08.994002 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 23:56:08.994179 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 23:56:08.994499 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
I20260806 23:56:08.996783 7442 heartbeater.cc:347] Connected to a master server at 127.10.225.190:45521
I20260806 23:56:08.997030 7442 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.225.130@KRBTEST.COM; defaulting to no policy
I20260806 23:56:08.997819 7442 heartbeater.cc:511] Master 127.10.225.190:45521 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.130@KRBTEST.COM" created.
I20260806 23:56:09.000155 7151 ts_manager.cc:194] Registered new tserver with Master: a23cdf8da0c144b192810ae3be6755c9 (127.10.225.129:45105)
I20260806 23:56:09.001351 7151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.129@KRBTEST.COM'} at 127.10.225.129:46315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
Entry for principal kudu/127.10.225.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab.
I20260806 23:56:09.016685 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:45521
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:09.128648 7449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:09.128930 7449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:09.128974 7449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:09.132990 7449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:09.133054 7449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:09.133114 7449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:56:09.138595 7449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.130.keytab
--principal=kudu/127.10.225.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:45521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:09.139813 7449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:09.140870 7449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:09.150435 7460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:09.150482 7449 server_base.cc:1034] running on GCE node
W20260806 23:56:09.150437 7459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:09.150449 7462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:09.151098 7449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:09.151798 7449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:09.152967 7449 hybrid_clock.cc:648] HybridClock initialized: now 1786060569152927 us; error 42 us; skew 500 ppm
Aug 06 23:56:09 dist-test-slave-nj21 krb5kdc[7088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060569, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:09.156832 7449 init.cc:377] Logged in from keytab as kudu/127.10.225.130@KRBTEST.COM (short username kudu)
I20260806 23:56:09.157795 7449 webserver.cc:460] Webserver started at http://127.10.225.130:46851/ using document root <none> and password file <none>
I20260806 23:56:09.158372 7449 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:09.158442 7449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:09.158648 7449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:09.160753 7449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "41dfec57b41441d1a4d26c956fd03b9e"
format_stamp: "Formatted at 2026-08-06 23:56:09 on dist-test-slave-nj21"
server_key: "8bab6937bb87c681dfa1f387d55bb68e"
server_key_iv: "afbe1873d68add19f9f11e37fcab73c5"
server_key_version: "encryptionkey@0"
I20260806 23:56:09.161295 7449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "41dfec57b41441d1a4d26c956fd03b9e"
format_stamp: "Formatted at 2026-08-06 23:56:09 on dist-test-slave-nj21"
server_key: "8bab6937bb87c681dfa1f387d55bb68e"
server_key_iv: "afbe1873d68add19f9f11e37fcab73c5"
server_key_version: "encryptionkey@0"
I20260806 23:56:09.164932 7449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 23:56:09.167335 7474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:09.168578 7449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:56:09.168687 7449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "41dfec57b41441d1a4d26c956fd03b9e"
format_stamp: "Formatted at 2026-08-06 23:56:09 on dist-test-slave-nj21"
server_key: "8bab6937bb87c681dfa1f387d55bb68e"
server_key_iv: "afbe1873d68add19f9f11e37fcab73c5"
server_key_version: "encryptionkey@0"
I20260806 23:56:09.168797 7449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:09.189693 7449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:09.192960 7449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:09.193145 7449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:09.193748 7449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:09.194598 7449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:09.194648 7449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:09.194703 7449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:09.194728 7449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:09.203577 7449 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:41295
I20260806 23:56:09.203604 7646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:41295 every 8 connection(s)
I20260806 23:56:09.204514 7449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
Aug 06 23:56:09 dist-test-slave-nj21 krb5kdc[7088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060569, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.130@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:09.214530 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 7449
I20260806 23:56:09.214637 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:09.214881 11142 external_mini_cluster.cc:1428] Setting key a181431d91adecabf58bd9adff719ca4
I20260806 23:56:09.218501 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.206255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:54087 (local address 127.10.225.190:45521)
0806 23:56:09.206433 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.206438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.207247 (+ 809us) server_negotiation.cc:407] Connection header received
0806 23:56:09.208600 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.208603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.208653 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.208730 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:09.209856 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.210383 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.211637 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.211971 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.212105 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.214811 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:09.214814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:09.214817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:09.214848 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:09.216411 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:09.216901 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:09.216905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:09.216907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:09.216995 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:09.217390 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:09.217394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:09.217396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:09.217605 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:09.217696 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.218199 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.218327 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 23:56:09.219359 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.206655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:45521 (local address 127.10.225.130:54087)
0806 23:56:09.207102 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:09.207144 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:09.208385 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:09.208870 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:09.208886 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:09.209225 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:09.209726 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.209739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.210520 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.210524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.211455 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.211465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.212135 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.212137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.212286 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.212874 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:09.212891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:09.214629 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:09.216528 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:09.216534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:09.216550 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:09.216805 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:09.217080 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:09.217083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:09.217085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:09.217223 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:09.217710 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:09.217717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:09.217970 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 23:56:09.218141 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 23:56:09.218500 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260806 23:56:09.220557 7647 heartbeater.cc:347] Connected to a master server at 127.10.225.190:45521
I20260806 23:56:09.220780 7647 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:09.221257 7647 heartbeater.cc:511] Master 127.10.225.190:45521 requested a full tablet report, sending...
I20260806 23:56:09.222297 7151 ts_manager.cc:194] Registered new tserver with Master: 41dfec57b41441d1a4d26c956fd03b9e (127.10.225.130:41295)
I20260806 23:56:09.222846 7151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.130@KRBTEST.COM'} at 127.10.225.130:54087
WARNING: no policy specified for kudu/127.10.225.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
Entry for principal kudu/127.10.225.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab.
I20260806 23:56:09.241959 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:45521
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:09.356074 7657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:09.356333 7657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:09.356377 7657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:09.360404 7657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:09.360462 7657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:09.360522 7657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:56:09.365749 7657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.131.keytab
--principal=kudu/127.10.225.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:45521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:09.366782 7657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:09.367789 7657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:56:09.377413 7657 server_base.cc:1034] running on GCE node
W20260806 23:56:09.377324 7664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:09.377353 7665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:09.377358 7667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:09.377960 7657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:09.378607 7657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:09.379738 7657 hybrid_clock.cc:648] HybridClock initialized: now 1786060569379727 us; error 30 us; skew 500 ppm
Aug 06 23:56:09 dist-test-slave-nj21 krb5kdc[7088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060569, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:09.382706 7657 init.cc:377] Logged in from keytab as kudu/127.10.225.131@KRBTEST.COM (short username kudu)
I20260806 23:56:09.383625 7657 webserver.cc:460] Webserver started at http://127.10.225.131:44755/ using document root <none> and password file <none>
I20260806 23:56:09.384222 7657 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:09.384297 7657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:09.384497 7657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:09.386574 7657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "b28c7aa651c94938952499dd57b49e84"
format_stamp: "Formatted at 2026-08-06 23:56:09 on dist-test-slave-nj21"
server_key: "fea674c26d244025f3d06451eea9877b"
server_key_iv: "e911dbabb272949ea66ba6ff16cec273"
server_key_version: "encryptionkey@0"
I20260806 23:56:09.387128 7657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "b28c7aa651c94938952499dd57b49e84"
format_stamp: "Formatted at 2026-08-06 23:56:09 on dist-test-slave-nj21"
server_key: "fea674c26d244025f3d06451eea9877b"
server_key_iv: "e911dbabb272949ea66ba6ff16cec273"
server_key_version: "encryptionkey@0"
I20260806 23:56:09.390995 7657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 23:56:09.393726 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:09.394966 7657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:56:09.395087 7657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "b28c7aa651c94938952499dd57b49e84"
format_stamp: "Formatted at 2026-08-06 23:56:09 on dist-test-slave-nj21"
server_key: "fea674c26d244025f3d06451eea9877b"
server_key_iv: "e911dbabb272949ea66ba6ff16cec273"
server_key_version: "encryptionkey@0"
I20260806 23:56:09.395191 7657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:09.411473 7657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:09.415184 7657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:09.415386 7657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:09.416052 7657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:09.416950 7657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:09.416999 7657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:09.417057 7657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:09.417081 7657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:09.426331 7657 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:46019
I20260806 23:56:09.426353 7845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:46019 every 8 connection(s)
I20260806 23:56:09.427220 7657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:56:09.429643 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 7657
I20260806 23:56:09.429780 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:56:09.430076 11142 external_mini_cluster.cc:1428] Setting key d48c5ee8470e6a0fd9fa4e7bc483ad51
Aug 06 23:56:09 dist-test-slave-nj21 krb5kdc[7088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060569, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.131@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:09.441151 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.429084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:53547 (local address 127.10.225.190:45521)
0806 23:56:09.429222 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.429225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.430063 (+ 838us) server_negotiation.cc:407] Connection header received
0806 23:56:09.431425 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.431428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.431480 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.431543 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:09.432811 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.433336 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.435023 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.435243 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.435374 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.437877 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:09.437881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:09.437883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:09.437907 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:09.439130 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:09.439717 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:09.439721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:09.439723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:09.439810 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:09.440151 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:09.440153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:09.440154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:09.440331 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:09.440408 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.440910 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.441023 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:56:09.441972 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.429422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:45521 (local address 127.10.225.131:53547)
0806 23:56:09.429899 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:09.429960 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:09.431243 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:09.431766 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:09.431775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:09.432213 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:09.432692 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.432706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.433590 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.433595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.434873 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.434888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.435439 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.435442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.435600 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.436222 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:09.436239 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:09.437698 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:09.439261 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:09.439275 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:09.439290 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:09.439575 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:09.439924 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:09.439927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:09.439928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:09.440074 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:09.440451 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:09.440457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:09.440699 (+ 242us) client_negotiation.cc:769] Sending connection context
0806 23:56:09.440872 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 23:56:09.441161 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
I20260806 23:56:09.443102 7847 heartbeater.cc:347] Connected to a master server at 127.10.225.190:45521
I20260806 23:56:09.443341 7847 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:09.443899 7847 heartbeater.cc:511] Master 127.10.225.190:45521 requested a full tablet report, sending...
I20260806 23:56:09.444875 7151 ts_manager.cc:194] Registered new tserver with Master: b28c7aa651c94938952499dd57b49e84 (127.10.225.131:46019)
I20260806 23:56:09.445411 7151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.225.131@KRBTEST.COM'} at 127.10.225.131:53547
I20260806 23:56:09.455776 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:56:09.464403 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.457785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.10.225.190:45521)
0806 23:56:09.457901 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.457905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.458012 (+ 107us) server_negotiation.cc:407] Connection header received
0806 23:56:09.458110 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.458112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.458153 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.458227 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:09.458940 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.459478 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.460362 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.460584 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.460667 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.461881 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:09.461885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:09.461887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:09.461909 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:09.463035 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:09.463391 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:09.463394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:09.463395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:09.463470 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:09.463660 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:09.463662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:09.463663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:09.463916 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:09.464027 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.464190 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.464281 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:56:09.467192 7151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:56:09.470304 7151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:09.485637 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.478887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.10.225.131:46019)
0806 23:56:09.479432 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.479450 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.480037 (+ 587us) server_negotiation.cc:407] Connection header received
0806 23:56:09.480289 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.480296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.480476 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.480571 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:09.481054 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.482259 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.484196 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.484863 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.484963 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.485125 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.485354 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.485465 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":120,"threads_started":1}
I20260806 23:56:09.485630 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.478711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:46019 (local address 127.0.0.1:41240)
0806 23:56:09.479909 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:09.479959 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:09.480112 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:09.480723 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:09.480725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:09.480738 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:09.480928 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.480934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.482417 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.482420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.484065 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.484075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.484998 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.485002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.485109 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.485156 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 23:56:09.485317 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 23:56:09.485465 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":151,"thread_start_us":87,"threads_started":1}
I20260806 23:56:09.486325 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.479490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:41295 (local address 127.0.0.1:48098)
0806 23:56:09.480162 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:09.480174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:09.480256 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:09.480694 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:09.480700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:09.480722 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:09.480926 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.480934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.483332 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.483336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.484253 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.484264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.485115 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.485119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.485203 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.485226 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:56:09.485316 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 23:56:09.486190 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":830,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260806 23:56:09.486465 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.479764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.10.225.130:41295)
0806 23:56:09.480199 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.480211 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.480241 (+ 30us) server_negotiation.cc:407] Connection header received
0806 23:56:09.480370 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.480374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.480533 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.480682 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:09.482283 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.483206 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.484401 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.484990 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.485115 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.485230 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.486210 (+ 980us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.486315 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":144,"threads_started":1}
I20260806 23:56:09.486712 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.478755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45105 (local address 127.0.0.1:59796)
0806 23:56:09.479911 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:09.479959 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:09.480098 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:09.480845 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:09.480848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:09.480864 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:09.480940 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.480945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.482417 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.482420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.484064 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.484075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.486309 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.486312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.486423 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.486445 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:56:09.486515 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 23:56:09.486591 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":299,"spinlock_wait_cycles":768,"thread_start_us":85,"threads_started":1}
I20260806 23:56:09.487260 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.479123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.10.225.129:45105)
0806 23:56:09.479587 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.479603 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.480239 (+ 636us) server_negotiation.cc:407] Connection header received
0806 23:56:09.480339 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.480345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.480534 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.480682 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:09.481072 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.482266 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.485187 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.486012 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.486125 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.486249 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.487025 (+ 776us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.487124 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":98,"threads_started":1}
I20260806 23:56:09.488847 7748 tablet_service.cc:1467] Processing CreateTablet for tablet 13710544b39c427a8518ff9fdd38f98e (DEFAULT_TABLE table=test-table [id=8e3b34980ebd4b08ad941a51fb65b342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:09.489359 7549 tablet_service.cc:1467] Processing CreateTablet for tablet 13710544b39c427a8518ff9fdd38f98e (DEFAULT_TABLE table=test-table [id=8e3b34980ebd4b08ad941a51fb65b342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:09.489993 7748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13710544b39c427a8518ff9fdd38f98e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:09.489856 7344 tablet_service.cc:1467] Processing CreateTablet for tablet 13710544b39c427a8518ff9fdd38f98e (DEFAULT_TABLE table=test-table [id=8e3b34980ebd4b08ad941a51fb65b342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:09.490563 7549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13710544b39c427a8518ff9fdd38f98e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:09.491092 7344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13710544b39c427a8518ff9fdd38f98e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:09.496310 7870 tablet_bootstrap.cc:492] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: Bootstrap starting.
I20260806 23:56:09.497449 7871 tablet_bootstrap.cc:492] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: Bootstrap starting.
I20260806 23:56:09.498322 7872 tablet_bootstrap.cc:492] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9: Bootstrap starting.
I20260806 23:56:09.498648 7870 tablet_bootstrap.cc:654] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:09.500124 7871 tablet_bootstrap.cc:654] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:09.500245 7872 tablet_bootstrap.cc:654] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:09.500491 7870 log.cc:824] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:09.501096 7872 log.cc:824] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:09.501327 7871 log.cc:824] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:09.502704 7870 tablet_bootstrap.cc:492] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: No bootstrap required, opened a new log
I20260806 23:56:09.502756 7872 tablet_bootstrap.cc:492] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9: No bootstrap required, opened a new log
I20260806 23:56:09.502921 7872 ts_tablet_manager.cc:1397] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 23:56:09.502923 7870 ts_tablet_manager.cc:1397] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 23:56:09.503268 7871 tablet_bootstrap.cc:492] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: No bootstrap required, opened a new log
I20260806 23:56:09.503458 7871 ts_tablet_manager.cc:1397] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 23:56:09.506783 7870 raft_consensus.cc:348] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } }
I20260806 23:56:09.506845 7872 raft_consensus.cc:348] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } }
I20260806 23:56:09.507117 7872 raft_consensus.cc:374] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:09.507052 7871 raft_consensus.cc:348] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } }
I20260806 23:56:09.507208 7872 raft_consensus.cc:729] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23cdf8da0c144b192810ae3be6755c9, State: Initialized, Role: FOLLOWER
I20260806 23:56:09.507306 7870 raft_consensus.cc:374] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:09.507337 7871 raft_consensus.cc:374] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:09.507414 7870 raft_consensus.cc:729] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28c7aa651c94938952499dd57b49e84, State: Initialized, Role: FOLLOWER
I20260806 23:56:09.507440 7871 raft_consensus.cc:729] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41dfec57b41441d1a4d26c956fd03b9e, State: Initialized, Role: FOLLOWER
I20260806 23:56:09.507771 7872 consensus_queue.cc:260] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } }
I20260806 23:56:09.507925 7870 consensus_queue.cc:260] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } }
I20260806 23:56:09.507925 7871 consensus_queue.cc:260] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } }
I20260806 23:56:09.508956 7870 ts_tablet_manager.cc:1428] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 23:56:09.508956 7871 ts_tablet_manager.cc:1428] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 23:56:09.508955 7872 ts_tablet_manager.cc:1428] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 23:56:09.509323 7442 heartbeater.cc:503] Master 127.10.225.190:45521 was elected leader, sending a full tablet report...
I20260806 23:56:09.509330 7647 heartbeater.cc:503] Master 127.10.225.190:45521 was elected leader, sending a full tablet report...
I20260806 23:56:09.510264 7847 heartbeater.cc:503] Master 127.10.225.190:45521 was elected leader, sending a full tablet report...
I20260806 23:56:09.556344 7879 raft_consensus.cc:482] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:09.556581 7879 raft_consensus.cc:504] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } }
I20260806 23:56:09.557971 7879 leader_election.cc:288] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b28c7aa651c94938952499dd57b49e84 (127.10.225.131:46019), 41dfec57b41441d1a4d26c956fd03b9e (127.10.225.130:41295)
I20260806 23:56:09.561861 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.558319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:35989 (local address 127.10.225.131:46019)
0806 23:56:09.558451 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.558455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.558677 (+ 222us) server_negotiation.cc:407] Connection header received
0806 23:56:09.558928 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.558930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.558977 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.559043 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:09.559329 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.559872 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.560924 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.561300 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.561412 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.561466 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.561667 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.561736 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:56:09.561889 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.558209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:46019 (local address 127.10.225.129:35989)
0806 23:56:09.558600 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:09.558614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:09.558723 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:09.559075 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:09.559078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:09.559096 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:09.559232 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.559237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.559975 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.559978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.560799 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.560808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.561416 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.561420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.561559 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.561578 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 23:56:09.561642 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 23:56:09.561713 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":38,"thread_start_us":108,"threads_started":1}
I20260806 23:56:09.561901 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.558735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:42947 (local address 127.10.225.130:41295)
0806 23:56:09.558865 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.558869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.558984 (+ 115us) server_negotiation.cc:407] Connection header received
0806 23:56:09.559091 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.559094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.559138 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.559200 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:09.559461 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.559977 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.560977 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.561377 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.561482 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.561533 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.561719 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.561787 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:56:09.561918 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.558573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:41295 (local address 127.10.225.129:42947)
0806 23:56:09.558895 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:09.558906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:09.559004 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:09.559221 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:09.559224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:09.559241 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:09.559359 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.559365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.560082 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.560085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.560856 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:09.560865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.561485 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.561489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:09.561607 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.561629 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 23:56:09.561690 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 23:56:09.561759 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260806 23:56:09.562534 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13710544b39c427a8518ff9fdd38f98e" candidate_uuid: "a23cdf8da0c144b192810ae3be6755c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28c7aa651c94938952499dd57b49e84" is_pre_election: true
I20260806 23:56:09.562536 7580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13710544b39c427a8518ff9fdd38f98e" candidate_uuid: "a23cdf8da0c144b192810ae3be6755c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41dfec57b41441d1a4d26c956fd03b9e" is_pre_election: true
I20260806 23:56:09.562829 7780 raft_consensus.cc:2454] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a23cdf8da0c144b192810ae3be6755c9 in term 0.
I20260806 23:56:09.562824 7580 raft_consensus.cc:2454] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a23cdf8da0c144b192810ae3be6755c9 in term 0.
I20260806 23:56:09.563283 7271 leader_election.cc:302] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a23cdf8da0c144b192810ae3be6755c9, b28c7aa651c94938952499dd57b49e84; no voters:
I20260806 23:56:09.563556 7879 raft_consensus.cc:2781] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:09.563635 7879 raft_consensus.cc:482] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:09.563690 7879 raft_consensus.cc:3037] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:09.564736 7879 raft_consensus.cc:504] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } }
I20260806 23:56:09.565073 7879 leader_election.cc:288] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [CANDIDATE]: Term 1 election: Requested vote from peers b28c7aa651c94938952499dd57b49e84 (127.10.225.131:46019), 41dfec57b41441d1a4d26c956fd03b9e (127.10.225.130:41295)
I20260806 23:56:09.565430 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13710544b39c427a8518ff9fdd38f98e" candidate_uuid: "a23cdf8da0c144b192810ae3be6755c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28c7aa651c94938952499dd57b49e84"
I20260806 23:56:09.565524 7580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13710544b39c427a8518ff9fdd38f98e" candidate_uuid: "a23cdf8da0c144b192810ae3be6755c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41dfec57b41441d1a4d26c956fd03b9e"
I20260806 23:56:09.565541 7780 raft_consensus.cc:3037] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:09.565650 7580 raft_consensus.cc:3037] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:09.566576 7780 raft_consensus.cc:2454] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a23cdf8da0c144b192810ae3be6755c9 in term 1.
I20260806 23:56:09.566718 7580 raft_consensus.cc:2454] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a23cdf8da0c144b192810ae3be6755c9 in term 1.
I20260806 23:56:09.566864 7271 leader_election.cc:302] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a23cdf8da0c144b192810ae3be6755c9, b28c7aa651c94938952499dd57b49e84; no voters:
I20260806 23:56:09.567037 7879 raft_consensus.cc:2781] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:09.567286 7879 raft_consensus.cc:686] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [term 1 LEADER]: Becoming Leader. State: Replica: a23cdf8da0c144b192810ae3be6755c9, State: Running, Role: LEADER
I20260806 23:56:09.567631 7879 consensus_queue.cc:237] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } }
I20260806 23:56:09.570731 7153 catalog_manager.cc:5495] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a23cdf8da0c144b192810ae3be6755c9 (127.10.225.129). New cstate: current_term: 1 leader_uuid: "a23cdf8da0c144b192810ae3be6755c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a23cdf8da0c144b192810ae3be6755c9" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 45105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 } health_report { overall_health: UNKNOWN } } }
I20260806 23:56:09.585454 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.581774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.10.225.129:45105)
0806 23:56:09.581903 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.581907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.581920 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:56:09.582005 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.582007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.582065 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.582151 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:09.582428 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.582958 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.583881 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.584130 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.584219 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.584588 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:09.585026 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:09.585123 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.585223 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.585281 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 23:56:09.591979 7580 raft_consensus.cc:1261] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [term 1 FOLLOWER]: Refusing update from remote peer a23cdf8da0c144b192810ae3be6755c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:09.592046 7780 raft_consensus.cc:1261] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [term 1 FOLLOWER]: Refusing update from remote peer a23cdf8da0c144b192810ae3be6755c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:09.592782 7887 consensus_queue.cc:1035] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41dfec57b41441d1a4d26c956fd03b9e" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:09.592994 7879 consensus_queue.cc:1035] T 13710544b39c427a8518ff9fdd38f98e P a23cdf8da0c144b192810ae3be6755c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b28c7aa651c94938952499dd57b49e84" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 46019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:09.603957 7894 mvcc.cc:205] Tried to move back new op lower bound from 7315704093039558656 to 7315704092951691264. Current Snapshot: MvccSnapshot[applied={T|T < 7315704093039558656}]
I20260806 23:56:09.611027 7154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:09.611192 7154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:09.613714 7154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c92bc644ef9c4038b678a11766596bb0: Sending DeleteTablet for 3 replicas of tablet 13710544b39c427a8518ff9fdd38f98e
I20260806 23:56:09.614562 7748 tablet_service.cc:1514] Processing DeleteTablet for tablet 13710544b39c427a8518ff9fdd38f98e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:09 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:41240
I20260806 23:56:09.614578 7549 tablet_service.cc:1514] Processing DeleteTablet for tablet 13710544b39c427a8518ff9fdd38f98e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:09 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:48098
I20260806 23:56:09.614715 7344 tablet_service.cc:1514] Processing DeleteTablet for tablet 13710544b39c427a8518ff9fdd38f98e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:09 UTC) from {username='kudu', principal='kudu/127.10.225.190@KRBTEST.COM'} at 127.0.0.1:59796
I20260806 23:56:09.615300 7905 tablet_replica.cc:330] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: stopping tablet replica
I20260806 23:56:09.615388 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 7251
I20260806 23:56:09.615630 7905 raft_consensus.cc:2229] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:09.615958 7905 raft_consensus.cc:2258] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:09.616291 7906 tablet_replica.cc:330] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: stopping tablet replica
I20260806 23:56:09.616539 7906 raft_consensus.cc:2229] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:09.616747 7906 raft_consensus.cc:2258] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:09.620827 7906 ts_tablet_manager.cc:1905] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:09.624034 7906 ts_tablet_manager.cc:1918] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:09.624169 7906 log.cc:1196] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/13710544b39c427a8518ff9fdd38f98e
I20260806 23:56:09.624549 7906 ts_tablet_manager.cc:1939] T 13710544b39c427a8518ff9fdd38f98e P 41dfec57b41441d1a4d26c956fd03b9e: Deleting consensus metadata
I20260806 23:56:09.625309 7905 ts_tablet_manager.cc:1905] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:09.625990 7134 catalog_manager.cc:4841] TS 41dfec57b41441d1a4d26c956fd03b9e (127.10.225.130:41295): tablet 13710544b39c427a8518ff9fdd38f98e (table test-table [id=8e3b34980ebd4b08ad941a51fb65b342]) successfully deleted
I20260806 23:56:09.627585 7905 ts_tablet_manager.cc:1918] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:09.627696 7905 log.cc:1196] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/13710544b39c427a8518ff9fdd38f98e
I20260806 23:56:09.627993 7905 ts_tablet_manager.cc:1939] T 13710544b39c427a8518ff9fdd38f98e P b28c7aa651c94938952499dd57b49e84: Deleting consensus metadata
W20260806 23:56:09.628991 7135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:45105 (error 108)
I20260806 23:56:09.629139 7134 catalog_manager.cc:4841] TS b28c7aa651c94938952499dd57b49e84 (127.10.225.131:46019): tablet 13710544b39c427a8518ff9fdd38f98e (table test-table [id=8e3b34980ebd4b08ad941a51fb65b342]) successfully deleted
I20260806 23:56:09.629899 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 7449
I20260806 23:56:09.630033 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.629746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:45105 (local address 127.0.0.1:59818)
0806 23:56:09.629874 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:09.629947 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 23:56:09.632992 7135 catalog_manager.cc:4584] TS a23cdf8da0c144b192810ae3be6755c9 (127.10.225.129:45105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13710544b39c427a8518ff9fdd38f98e: Network error: Client connection negotiation failed: client connection to 127.10.225.129:45105: connect: Connection refused (error 111)
I20260806 23:56:09.642676 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 7657
I20260806 23:56:09.662297 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 7105
2026-08-06T23:56:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2455 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T23:56:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:09Z Disabled control of system clock
I20260806 23:56:09.699837 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41967
--webserver_interface=127.10.225.190
--webserver_port=0
--builtin_ntp_servers=127.10.225.148:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:41967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 23:56:09.814379 7916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:09.814649 7916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:09.814685 7916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:09.818768 7916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:09.818835 7916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:09.818849 7916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:09.818866 7916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:09.818881 7916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:09.823721 7916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:41967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:41967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:09.824918 7916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:09.826414 7916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:09.834981 7926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:09.835069 7928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:09.835084 7916 server_base.cc:1034] running on GCE node
W20260806 23:56:09.834992 7925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:09.835899 7916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:09.836982 7916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:09.838157 7916 hybrid_clock.cc:648] HybridClock initialized: now 1786060569838132 us; error 37 us; skew 500 ppm
I20260806 23:56:09.844370 7916 webserver.cc:460] Webserver started at http://127.10.225.190:33959/ using document root <none> and password file <none>
I20260806 23:56:09.844988 7916 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:09.845063 7916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:09.845271 7916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:09.847414 7916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "d21771d979f744079a13b9f7700f8636"
format_stamp: "Formatted at 2026-08-06 23:56:09 on dist-test-slave-nj21"
server_key: "feb4c67375408f6f8dc400fc779cfb25"
server_key_iv: "27bb14739dad12d8344426ae5eb7315c"
server_key_version: "encryptionkey@0"
I20260806 23:56:09.848011 7916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "d21771d979f744079a13b9f7700f8636"
format_stamp: "Formatted at 2026-08-06 23:56:09 on dist-test-slave-nj21"
server_key: "feb4c67375408f6f8dc400fc779cfb25"
server_key_iv: "27bb14739dad12d8344426ae5eb7315c"
server_key_version: "encryptionkey@0"
I20260806 23:56:09.851894 7916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 23:56:09.854449 7935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:09.855747 7916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:09.855901 7916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "d21771d979f744079a13b9f7700f8636"
format_stamp: "Formatted at 2026-08-06 23:56:09 on dist-test-slave-nj21"
server_key: "feb4c67375408f6f8dc400fc779cfb25"
server_key_iv: "27bb14739dad12d8344426ae5eb7315c"
server_key_version: "encryptionkey@0"
I20260806 23:56:09.856036 7916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:09.872375 7916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:09.873142 7916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:09.873461 7916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:09.880728 7916 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:41967
I20260806 23:56:09.880746 8008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:41967 every 8 connection(s)
I20260806 23:56:09.881817 7916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:09.884985 8009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:09.886497 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 7916
I20260806 23:56:09.886627 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:09.886900 11142 external_mini_cluster.cc:1428] Setting key d49eec595f6aa545a7ee2ad65db6d10f
I20260806 23:56:09.891206 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636: Bootstrap starting.
I20260806 23:56:09.893455 8009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:09.894168 8009 log.cc:824] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:09.896015 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:09.888323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.10.225.190:41967)
0806 23:56:09.888980 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:09.888995 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:09.889030 (+ 35us) server_negotiation.cc:407] Connection header received
0806 23:56:09.889701 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:09.889718 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:09.890147 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:09.890443 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:09.890869 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.891908 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.892768 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:09.893244 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:09.893391 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:09.894162 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:09.894167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:09.894180 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:09.894618 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:09.894687 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:09.894715 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:09.894815 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:09.894967 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 23:56:09.895231 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":132,"threads_started":1}
I20260806 23:56:09.896230 8009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636: No bootstrap required, opened a new log
I20260806 23:56:09.899950 8009 raft_consensus.cc:348] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21771d979f744079a13b9f7700f8636" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41967 } }
I20260806 23:56:09.900182 8009 raft_consensus.cc:374] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:09.900238 8009 raft_consensus.cc:729] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21771d979f744079a13b9f7700f8636, State: Initialized, Role: FOLLOWER
I20260806 23:56:09.900662 8009 consensus_queue.cc:260] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21771d979f744079a13b9f7700f8636" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41967 } }
I20260806 23:56:09.900804 8009 raft_consensus.cc:388] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:09.900877 8009 raft_consensus.cc:482] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:09.900975 8009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:09.901929 8009 raft_consensus.cc:504] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21771d979f744079a13b9f7700f8636" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41967 } }
I20260806 23:56:09.902259 8009 leader_election.cc:302] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d21771d979f744079a13b9f7700f8636; no voters:
I20260806 23:56:09.902581 8009 leader_election.cc:288] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:09.902684 8024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:09.902928 8024 raft_consensus.cc:686] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [term 1 LEADER]: Becoming Leader. State: Replica: d21771d979f744079a13b9f7700f8636, State: Running, Role: LEADER
I20260806 23:56:09.903258 8024 consensus_queue.cc:237] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21771d979f744079a13b9f7700f8636" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41967 } }
I20260806 23:56:09.903941 8009 sys_catalog.cc:564] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:09.904984 8027 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d21771d979f744079a13b9f7700f8636. Latest consensus state: current_term: 1 leader_uuid: "d21771d979f744079a13b9f7700f8636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21771d979f744079a13b9f7700f8636" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41967 } } }
I20260806 23:56:09.905113 8027 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:09.905350 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d21771d979f744079a13b9f7700f8636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21771d979f744079a13b9f7700f8636" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 41967 } } }
I20260806 23:56:09.905556 8040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:09.907780 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:09.908913 8040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:09.914108 8040 catalog_manager.cc:1347] Generated new cluster ID: c17bd33747d6435ca0bc4d732c589a30
I20260806 23:56:09.914178 8040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:09.930094 8040 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:09.931085 8040 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:09.938402 8040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636: Generated new TSK 0
I20260806 23:56:09.939232 8040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:09.952823 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.129:0
--local_ip_for_outbound_sockets=127.10.225.129
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41967
--builtin_ntp_servers=127.10.225.148:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 23:56:10.068009 8057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:10.068316 8057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:10.068375 8057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:10.072526 8057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:10.072590 8057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:10.072670 8057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.129
I20260806 23:56:10.077605 8057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.225.129
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.129
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:10.078778 8057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:10.080209 8057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:10.089649 8067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:10.089640 8068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:10.089636 8072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:10.089880 8057 server_base.cc:1034] running on GCE node
I20260806 23:56:10.090430 8057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:10.091055 8057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:10.092206 8057 hybrid_clock.cc:648] HybridClock initialized: now 1786060570092191 us; error 26 us; skew 500 ppm
I20260806 23:56:10.093981 8057 webserver.cc:460] Webserver started at http://127.10.225.129:46251/ using document root <none> and password file <none>
I20260806 23:56:10.094525 8057 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:10.094583 8057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:10.094755 8057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:10.097155 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/data/instance:
uuid: "1f42572316994f19b4e7d17941c645a8"
format_stamp: "Formatted at 2026-08-06 23:56:10 on dist-test-slave-nj21"
server_key: "a654d95154b3d3aa0928ddfb0f2f343a"
server_key_iv: "79c7d257a9d419353d55cd2adec1bb71"
server_key_version: "encryptionkey@0"
I20260806 23:56:10.097713 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance:
uuid: "1f42572316994f19b4e7d17941c645a8"
format_stamp: "Formatted at 2026-08-06 23:56:10 on dist-test-slave-nj21"
server_key: "a654d95154b3d3aa0928ddfb0f2f343a"
server_key_iv: "79c7d257a9d419353d55cd2adec1bb71"
server_key_version: "encryptionkey@0"
I20260806 23:56:10.101523 8057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 23:56:10.104089 8081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:10.105512 8057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:10.105625 8057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/wal
uuid: "1f42572316994f19b4e7d17941c645a8"
format_stamp: "Formatted at 2026-08-06 23:56:10 on dist-test-slave-nj21"
server_key: "a654d95154b3d3aa0928ddfb0f2f343a"
server_key_iv: "79c7d257a9d419353d55cd2adec1bb71"
server_key_version: "encryptionkey@0"
I20260806 23:56:10.105734 8057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:10.132145 8057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:10.132962 8057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:10.133167 8057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:10.133854 8057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:10.134929 8057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:10.134994 8057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:10.135046 8057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:10.135079 8057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:10.144790 8057 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.129:35883
I20260806 23:56:10.144843 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.129:35883 every 8 connection(s)
I20260806 23:56:10.145736 8057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/data/info.pb
I20260806 23:56:10.150911 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 8057
I20260806 23:56:10.151180 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-0/wal/instance
I20260806 23:56:10.151647 11142 external_mini_cluster.cc:1428] Setting key 8c7ef37b7e99f9802302f7d125051e10
I20260806 23:56:10.154145 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.130:0
--local_ip_for_outbound_sockets=127.10.225.130
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41967
--builtin_ntp_servers=127.10.225.148:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 23:56:10.156772 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.147645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:51427 (local address 127.10.225.190:41967)
0806 23:56:10.147828 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.147833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.148943 (+ 1110us) server_negotiation.cc:407] Connection header received
0806 23:56:10.150358 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.150362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.150416 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.150496 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:10.152152 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.152788 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.154354 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.154631 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.154702 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.156201 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:10.156205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:10.156208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:10.156224 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:10.156251 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:10.156261 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:10.156343 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.156550 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.156620 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 23:56:10.158025 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.148105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41967 (local address 127.10.225.129:51427)
0806 23:56:10.148793 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.148841 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.150128 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.151022 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.151052 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.151185 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:10.151965 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.151993 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.152948 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.152951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.154198 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.154212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.154923 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.154927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.155123 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.155797 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:56:10.155799 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:10.155899 (+ 100us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:56:10.156005 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:10.156387 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:10.156398 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:10.156404 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.156522 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.156774 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":129,"threads_started":1}
I20260806 23:56:10.159579 8265 heartbeater.cc:347] Connected to a master server at 127.10.225.190:41967
I20260806 23:56:10.159889 8265 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:10.160665 8265 heartbeater.cc:511] Master 127.10.225.190:41967 requested a full tablet report, sending...
I20260806 23:56:10.162345 7957 ts_manager.cc:194] Registered new tserver with Master: 1f42572316994f19b4e7d17941c645a8 (127.10.225.129:35883)
I20260806 23:56:10.163511 7957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.129:51427
W20260806 23:56:10.272835 8275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:10.273118 8275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:10.273164 8275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:10.277478 8275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:10.277558 8275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:10.277626 8275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.130
I20260806 23:56:10.282562 8275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.225.130
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.130
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:10.283703 8275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:10.285090 8275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:10.294811 8284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:10.294838 8282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:10.294814 8286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:10.295018 8275 server_base.cc:1034] running on GCE node
I20260806 23:56:10.295579 8275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:10.296258 8275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:10.297423 8275 hybrid_clock.cc:648] HybridClock initialized: now 1786060570297407 us; error 29 us; skew 500 ppm
I20260806 23:56:10.299325 8275 webserver.cc:460] Webserver started at http://127.10.225.130:33423/ using document root <none> and password file <none>
I20260806 23:56:10.299954 8275 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:10.300028 8275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:10.300242 8275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:10.302402 8275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/data/instance:
uuid: "95665a79758b4269b8d3ab90bb11745a"
format_stamp: "Formatted at 2026-08-06 23:56:10 on dist-test-slave-nj21"
server_key: "32ac47354f194de7908476ae23c0a558"
server_key_iv: "b650989418e4524f5d5ecd258a16a01c"
server_key_version: "encryptionkey@0"
I20260806 23:56:10.302960 8275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance:
uuid: "95665a79758b4269b8d3ab90bb11745a"
format_stamp: "Formatted at 2026-08-06 23:56:10 on dist-test-slave-nj21"
server_key: "32ac47354f194de7908476ae23c0a558"
server_key_iv: "b650989418e4524f5d5ecd258a16a01c"
server_key_version: "encryptionkey@0"
I20260806 23:56:10.307211 8275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 23:56:10.309867 8294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:10.311159 8275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:10.311280 8275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/wal
uuid: "95665a79758b4269b8d3ab90bb11745a"
format_stamp: "Formatted at 2026-08-06 23:56:10 on dist-test-slave-nj21"
server_key: "32ac47354f194de7908476ae23c0a558"
server_key_iv: "b650989418e4524f5d5ecd258a16a01c"
server_key_version: "encryptionkey@0"
I20260806 23:56:10.311400 8275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:10.333770 8275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:10.334578 8275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:10.334789 8275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:10.335435 8275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:10.336549 8275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:10.336604 8275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:10.336652 8275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:10.336676 8275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:10.346169 8275 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.130:42999
I20260806 23:56:10.346196 8461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.130:42999 every 8 connection(s)
I20260806 23:56:10.347103 8275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/data/info.pb
I20260806 23:56:10.352274 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 8275
I20260806 23:56:10.352407 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/wal/instance
I20260806 23:56:10.352695 11142 external_mini_cluster.cc:1428] Setting key 18866d1f653367cdbaae5c8409ea8f72
I20260806 23:56:10.354696 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.225.131:0
--local_ip_for_outbound_sockets=127.10.225.131
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41967
--builtin_ntp_servers=127.10.225.148:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 23:56:10.356806 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.349033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:38845 (local address 127.10.225.190:41967)
0806 23:56:10.349185 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.349189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.350238 (+ 1049us) server_negotiation.cc:407] Connection header received
0806 23:56:10.351609 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.351612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.351662 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.351785 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:10.352686 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.353344 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.354746 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.355043 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.355122 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.356326 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:10.356329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:10.356331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:10.356343 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:10.356363 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:10.356372 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:10.356426 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.356599 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.356660 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 23:56:10.357647 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.349428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41967 (local address 127.10.225.130:38845)
0806 23:56:10.350099 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.350139 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.351397 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.351924 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.351930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.352011 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:10.352550 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.352565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.353461 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.353464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.354625 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.354638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.355187 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.355190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.355382 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.356044 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:56:10.356046 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:10.356098 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:56:10.356180 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:10.356444 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:10.356453 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:10.356460 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.356592 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.356848 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":116,"threads_started":1}
I20260806 23:56:10.358875 8462 heartbeater.cc:347] Connected to a master server at 127.10.225.190:41967
I20260806 23:56:10.359148 8462 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:10.360255 8462 heartbeater.cc:511] Master 127.10.225.190:41967 requested a full tablet report, sending...
I20260806 23:56:10.362080 7957 ts_manager.cc:194] Registered new tserver with Master: 95665a79758b4269b8d3ab90bb11745a (127.10.225.130:42999)
I20260806 23:56:10.362663 7957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.130:38845
W20260806 23:56:10.477614 8470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:10.477913 8470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:10.477974 8470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:10.483101 8470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:10.483175 8470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:10.483260 8470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.225.131
I20260806 23:56:10.488569 8470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.225.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.225.131
--webserver_port=0
--tserver_master_addrs=127.10.225.190:41967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.225.131
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:10.489768 8470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:10.491209 8470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:10.501946 8480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:10.502039 8470 server_base.cc:1034] running on GCE node
W20260806 23:56:10.501956 8478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:10.501950 8477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:10.502660 8470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:10.503352 8470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:10.504669 8470 hybrid_clock.cc:648] HybridClock initialized: now 1786060570504670 us; error 98 us; skew 500 ppm
I20260806 23:56:10.506721 8470 webserver.cc:460] Webserver started at http://127.10.225.131:32893/ using document root <none> and password file <none>
I20260806 23:56:10.507306 8470 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:10.507386 8470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:10.507598 8470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:10.510280 8470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/data/instance:
uuid: "c46eebf1c2cf44c888c3be0b593f769a"
format_stamp: "Formatted at 2026-08-06 23:56:10 on dist-test-slave-nj21"
server_key: "7db3a284f408fc1c7fd6d35092d73d47"
server_key_iv: "70e88fe8cf863f263887d2f74e069318"
server_key_version: "encryptionkey@0"
I20260806 23:56:10.511029 8470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance:
uuid: "c46eebf1c2cf44c888c3be0b593f769a"
format_stamp: "Formatted at 2026-08-06 23:56:10 on dist-test-slave-nj21"
server_key: "7db3a284f408fc1c7fd6d35092d73d47"
server_key_iv: "70e88fe8cf863f263887d2f74e069318"
server_key_version: "encryptionkey@0"
I20260806 23:56:10.515988 8470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 23:56:10.518653 8490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:10.520481 8470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:10.520623 8470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/wal
uuid: "c46eebf1c2cf44c888c3be0b593f769a"
format_stamp: "Formatted at 2026-08-06 23:56:10 on dist-test-slave-nj21"
server_key: "7db3a284f408fc1c7fd6d35092d73d47"
server_key_iv: "70e88fe8cf863f263887d2f74e069318"
server_key_version: "encryptionkey@0"
I20260806 23:56:10.520741 8470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:10.531848 8470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:10.532552 8470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:10.532729 8470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:10.533488 8470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 23:56:10.534695 8470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 23:56:10.534750 8470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:10.534807 8470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 23:56:10.534837 8470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:10.545207 8470 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.131:38869
I20260806 23:56:10.545241 8652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.131:38869 every 8 connection(s)
I20260806 23:56:10.546165 8470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/data/info.pb
I20260806 23:56:10.553334 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 8470
I20260806 23:56:10.553481 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/wal/instance
I20260806 23:56:10.553813 11142 external_mini_cluster.cc:1428] Setting key 579988aede22d63655fcf97ab8fd176d
I20260806 23:56:10.556690 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.548468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.131:36649 (local address 127.10.225.190:41967)
0806 23:56:10.548621 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.548626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.549815 (+ 1189us) server_negotiation.cc:407] Connection header received
0806 23:56:10.551246 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.551249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.551323 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.551404 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:10.552427 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.553096 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.554621 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.554909 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.554999 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.556235 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:10.556239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:10.556242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:10.556255 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:10.556277 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:10.556284 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:10.556343 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.556489 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.556550 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 23:56:10.557552 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.548908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:41967 (local address 127.10.225.131:36649)
0806 23:56:10.549645 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.549687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.550986 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.551582 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.551589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.551696 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:10.552218 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.552230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.553265 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.553275 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.554510 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.554522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.555066 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.555070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.555241 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.555968 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 23:56:10.555970 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:10.556023 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 23:56:10.556098 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:10.556352 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:10.556361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:10.556367 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.556483 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.556766 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
I20260806 23:56:10.558969 8653 heartbeater.cc:347] Connected to a master server at 127.10.225.190:41967
I20260806 23:56:10.559198 8653 heartbeater.cc:464] Registering TS with master...
I20260806 23:56:10.559832 8653 heartbeater.cc:511] Master 127.10.225.190:41967 requested a full tablet report, sending...
I20260806 23:56:10.560894 7957 ts_manager.cc:194] Registered new tserver with Master: c46eebf1c2cf44c888c3be0b593f769a (127.10.225.131:38869)
I20260806 23:56:10.561450 7957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.225.131:36649
I20260806 23:56:10.567952 11142 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 23:56:10.574520 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.570274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.10.225.190:41967)
0806 23:56:10.570410 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.570413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.570688 (+ 275us) server_negotiation.cc:407] Connection header received
0806 23:56:10.570810 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.570813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.570863 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.570923 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:10.571209 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.571907 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.572732 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.572957 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.573044 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.574046 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:10.574049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:10.574051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:10.574068 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:10.574090 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:10.574098 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:10.574178 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.574309 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.574375 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 23:56:10.578048 7957 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 23:56:10.581252 7957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 23:56:10.595985 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.590290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.10.225.131:38869)
0806 23:56:10.590780 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.590792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.590817 (+ 25us) server_negotiation.cc:407] Connection header received
0806 23:56:10.590892 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.590897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.591115 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.591275 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:10.591579 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.592713 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.594664 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.595252 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.595379 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.595619 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.595708 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.595812 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":172,"threads_started":1}
I20260806 23:56:10.595980 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.589732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:38869 (local address 127.0.0.1:33378)
0806 23:56:10.590468 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.590480 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.590545 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.591276 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.591282 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.591307 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:10.591459 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.591466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.592820 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.592822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.594502 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.594516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.595375 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.595379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.595484 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.595522 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.595629 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.595789 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":122,"thread_start_us":118,"threads_started":1}
I20260806 23:56:10.596521 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.590461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:35883 (local address 127.0.0.1:53838)
0806 23:56:10.591259 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.591272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.591378 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.592051 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.592054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.592074 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:10.592168 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.592175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.593295 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.593299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.594377 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.594404 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.596102 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.596107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.596222 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.596246 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.596320 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.596385 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"spinlock_wait_cycles":896,"thread_start_us":77,"threads_started":1}
I20260806 23:56:10.596670 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.590892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.10.225.129:35883)
0806 23:56:10.591608 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.591620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.591642 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:56:10.591735 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.591739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.591907 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.592047 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:10.592285 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.593168 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.594685 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.595702 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.595858 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.596027 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.596390 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.596504 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":113,"threads_started":1}
I20260806 23:56:10.597144 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.589732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:42999 (local address 127.0.0.1:47346)
0806 23:56:10.590135 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.590424 (+ 289us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.590545 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.591459 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.591462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.591478 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:10.591564 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.591570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.592753 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.592756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.594665 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.594674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.596490 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.596494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.596910 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.596939 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.596991 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.597047 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":455,"thread_start_us":102,"threads_started":1}
I20260806 23:56:10.597982 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.589907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.10.225.130:42999)
0806 23:56:10.590846 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.590873 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.590895 (+ 22us) server_negotiation.cc:407] Connection header received
0806 23:56:10.590971 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.590976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.591245 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.591398 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:10.591688 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.592643 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.594817 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.595829 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.595944 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.597584 (+ 1640us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.597659 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.597786 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":118,"threads_started":1}
I20260806 23:56:10.600030 8557 tablet_service.cc:1467] Processing CreateTablet for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3 (DEFAULT_TABLE table=test-table [id=d2cb8a96f73b46779dc12fbae02594c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:10.600765 8158 tablet_service.cc:1467] Processing CreateTablet for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3 (DEFAULT_TABLE table=test-table [id=d2cb8a96f73b46779dc12fbae02594c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:10.601392 8557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:10.601596 8158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:10.601356 8370 tablet_service.cc:1467] Processing CreateTablet for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3 (DEFAULT_TABLE table=test-table [id=d2cb8a96f73b46779dc12fbae02594c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 23:56:10.602340 8370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:10.608385 8681 tablet_bootstrap.cc:492] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8: Bootstrap starting.
I20260806 23:56:10.610030 8682 tablet_bootstrap.cc:492] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: Bootstrap starting.
I20260806 23:56:10.610733 8681 tablet_bootstrap.cc:654] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:10.611454 8681 log.cc:824] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:10.611816 8682 tablet_bootstrap.cc:654] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:10.612032 8684 tablet_bootstrap.cc:492] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: Bootstrap starting.
I20260806 23:56:10.612638 8682 log.cc:824] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:10.614467 8682 tablet_bootstrap.cc:492] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: No bootstrap required, opened a new log
I20260806 23:56:10.614666 8682 ts_tablet_manager.cc:1397] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 23:56:10.614826 8684 tablet_bootstrap.cc:654] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:10.614850 8681 tablet_bootstrap.cc:492] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8: No bootstrap required, opened a new log
I20260806 23:56:10.615046 8681 ts_tablet_manager.cc:1397] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 23:56:10.615777 8684 log.cc:824] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:10.618101 8684 tablet_bootstrap.cc:492] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: No bootstrap required, opened a new log
I20260806 23:56:10.618448 8684 ts_tablet_manager.cc:1397] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 23:56:10.619328 8682 raft_consensus.cc:348] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.619601 8682 raft_consensus.cc:374] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:10.619659 8682 raft_consensus.cc:729] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46eebf1c2cf44c888c3be0b593f769a, State: Initialized, Role: FOLLOWER
I20260806 23:56:10.620182 8682 consensus_queue.cc:260] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.620904 8653 heartbeater.cc:503] Master 127.10.225.190:41967 was elected leader, sending a full tablet report...
I20260806 23:56:10.620980 8681 raft_consensus.cc:348] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.621410 8681 raft_consensus.cc:374] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:10.621483 8681 raft_consensus.cc:729] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f42572316994f19b4e7d17941c645a8, State: Initialized, Role: FOLLOWER
I20260806 23:56:10.621941 8681 consensus_queue.cc:260] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.623485 8682 ts_tablet_manager.cc:1428] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 23:56:10.624258 8684 raft_consensus.cc:348] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.624538 8684 raft_consensus.cc:374] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:10.624614 8684 raft_consensus.cc:729] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95665a79758b4269b8d3ab90bb11745a, State: Initialized, Role: FOLLOWER
I20260806 23:56:10.625015 8681 ts_tablet_manager.cc:1428] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260806 23:56:10.625065 8684 consensus_queue.cc:260] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.625379 8265 heartbeater.cc:503] Master 127.10.225.190:41967 was elected leader, sending a full tablet report...
I20260806 23:56:10.626461 8462 heartbeater.cc:503] Master 127.10.225.190:41967 was elected leader, sending a full tablet report...
I20260806 23:56:10.627781 8684 ts_tablet_manager.cc:1428] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
W20260806 23:56:10.646929 8267 tablet.cc:2369] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:10.762648 8694 raft_consensus.cc:482] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:10.762883 8694 raft_consensus.cc:504] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.764096 8694 leader_election.cc:288] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c46eebf1c2cf44c888c3be0b593f769a (127.10.225.131:38869), 1f42572316994f19b4e7d17941c645a8 (127.10.225.129:35883)
I20260806 23:56:10.764472 8692 raft_consensus.cc:482] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 23:56:10.764670 8692 raft_consensus.cc:504] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.766485 8692 leader_election.cc:288] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95665a79758b4269b8d3ab90bb11745a (127.10.225.130:42999), c46eebf1c2cf44c888c3be0b593f769a (127.10.225.131:38869)
I20260806 23:56:10.769094 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.764388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:38869 (local address 127.10.225.130:59715)
0806 23:56:10.764545 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.764563 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.764680 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.765155 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.765158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.765182 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:10.765321 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.765329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.766317 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.766321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.767599 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.767610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.768652 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.768655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.768752 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.768789 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.768860 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.768940 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":82}
I20260806 23:56:10.769622 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.764815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:42197 (local address 127.10.225.129:35883)
0806 23:56:10.765063 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.765068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.765180 (+ 112us) server_negotiation.cc:407] Connection header received
0806 23:56:10.765351 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.765354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.765401 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.765461 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:10.765734 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.766412 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.768329 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.768920 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.769023 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.769067 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.769408 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.769504 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":167}
I20260806 23:56:10.769806 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.764500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.130:59715 (local address 127.10.225.131:38869)
0806 23:56:10.764616 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.764621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.764636 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:56:10.764917 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.764920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.764976 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.765052 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:10.765450 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.766187 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.767736 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.768360 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.768454 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.768507 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.769569 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.769668 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50}
I20260806 23:56:10.770040 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.764674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:35883 (local address 127.10.225.130:42197)
0806 23:56:10.765102 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.765116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.765243 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.765487 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.765490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.765508 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:10.765623 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.765629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.766902 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.766905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.768189 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.768199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.769182 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.769185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.769283 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.769304 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.769803 (+ 499us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.769879 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":37,"spinlock_wait_cycles":896,"thread_start_us":154,"threads_started":1}
I20260806 23:56:10.770422 8586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5a70cd9ae34c1ba5dd09f57ecf9df3" candidate_uuid: "95665a79758b4269b8d3ab90bb11745a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c46eebf1c2cf44c888c3be0b593f769a" is_pre_election: true
I20260806 23:56:10.770612 8188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5a70cd9ae34c1ba5dd09f57ecf9df3" candidate_uuid: "95665a79758b4269b8d3ab90bb11745a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f42572316994f19b4e7d17941c645a8" is_pre_election: true
I20260806 23:56:10.770746 8586 raft_consensus.cc:2454] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95665a79758b4269b8d3ab90bb11745a in term 0.
I20260806 23:56:10.770835 8188 raft_consensus.cc:2454] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95665a79758b4269b8d3ab90bb11745a in term 0.
I20260806 23:56:10.771241 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.766825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:48995 (local address 127.10.225.130:42999)
0806 23:56:10.767017 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.767021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.767032 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:56:10.767068 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.767070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.767109 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.767173 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:10.767555 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.768272 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.770068 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.770605 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.770692 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.770734 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.771073 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.771144 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260806 23:56:10.771297 8298 leader_election.cc:302] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95665a79758b4269b8d3ab90bb11745a, c46eebf1c2cf44c888c3be0b593f769a; no voters:
I20260806 23:56:10.771481 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.766057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.131:38869 (local address 127.10.225.129:42985)
0806 23:56:10.766608 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.766620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.766698 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.767191 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.767194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.767213 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:10.767346 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.767353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.768487 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.768490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.769968 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.769987 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.770991 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.770995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.771104 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.771128 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.771178 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.771245 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":14,"thread_start_us":125,"threads_started":1}
I20260806 23:56:10.771855 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.766541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.225.129:42985 (local address 127.10.225.131:38869)
0806 23:56:10.766975 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.766982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.766994 (+ 12us) server_negotiation.cc:407] Connection header received
0806 23:56:10.767032 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.767035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.767074 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.767447 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 23:56:10.767550 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.768366 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.770171 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.770800 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.771583 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.771625 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.771659 (+ 34us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.771749 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":363,"thread_start_us":109,"threads_started":1}
I20260806 23:56:10.771845 8694 raft_consensus.cc:2781] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:10.771929 8694 raft_consensus.cc:482] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:10.771976 8694 raft_consensus.cc:3037] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:10.772078 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.766138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.130:42999 (local address 127.10.225.129:48995)
0806 23:56:10.766745 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.766759 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:10.766854 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:10.767264 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:10.767266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:10.767285 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 23:56:10.767416 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.767423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.768407 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.768410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.769914 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:10.769924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.770827 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.770831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:10.770934 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.770957 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 23:56:10.771348 (+ 391us) client_negotiation.cc:240] Negotiation successful
0806 23:56:10.771425 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":113,"spinlock_wait_cycles":57216,"thread_start_us":146,"threads_started":1}
I20260806 23:56:10.772219 8586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5a70cd9ae34c1ba5dd09f57ecf9df3" candidate_uuid: "1f42572316994f19b4e7d17941c645a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c46eebf1c2cf44c888c3be0b593f769a" is_pre_election: true
I20260806 23:56:10.772346 8586 raft_consensus.cc:2454] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f42572316994f19b4e7d17941c645a8 in term 0.
I20260806 23:56:10.772625 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5a70cd9ae34c1ba5dd09f57ecf9df3" candidate_uuid: "1f42572316994f19b4e7d17941c645a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95665a79758b4269b8d3ab90bb11745a" is_pre_election: true
I20260806 23:56:10.772696 8086 leader_election.cc:302] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f42572316994f19b4e7d17941c645a8, c46eebf1c2cf44c888c3be0b593f769a; no voters:
I20260806 23:56:10.772980 8692 raft_consensus.cc:2781] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 23:56:10.773072 8692 raft_consensus.cc:482] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 23:56:10.773103 8692 raft_consensus.cc:3037] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:10.773356 8694 raft_consensus.cc:504] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.773599 8399 raft_consensus.cc:2379] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f42572316994f19b4e7d17941c645a8 in current term 1: Already voted for candidate 95665a79758b4269b8d3ab90bb11745a in this term.
I20260806 23:56:10.773726 8694 leader_election.cc:288] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [CANDIDATE]: Term 1 election: Requested vote from peers c46eebf1c2cf44c888c3be0b593f769a (127.10.225.131:38869), 1f42572316994f19b4e7d17941c645a8 (127.10.225.129:35883)
I20260806 23:56:10.774199 8188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5a70cd9ae34c1ba5dd09f57ecf9df3" candidate_uuid: "95665a79758b4269b8d3ab90bb11745a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f42572316994f19b4e7d17941c645a8"
I20260806 23:56:10.774228 8586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5a70cd9ae34c1ba5dd09f57ecf9df3" candidate_uuid: "95665a79758b4269b8d3ab90bb11745a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c46eebf1c2cf44c888c3be0b593f769a"
I20260806 23:56:10.774438 8586 raft_consensus.cc:3037] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:10.774528 8692 raft_consensus.cc:504] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.774907 8188 raft_consensus.cc:2379] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 95665a79758b4269b8d3ab90bb11745a in current term 1: Already voted for candidate 1f42572316994f19b4e7d17941c645a8 in this term.
I20260806 23:56:10.775056 8692 leader_election.cc:288] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [CANDIDATE]: Term 1 election: Requested vote from peers 95665a79758b4269b8d3ab90bb11745a (127.10.225.130:42999), c46eebf1c2cf44c888c3be0b593f769a (127.10.225.131:38869)
I20260806 23:56:10.775485 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5a70cd9ae34c1ba5dd09f57ecf9df3" candidate_uuid: "1f42572316994f19b4e7d17941c645a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c46eebf1c2cf44c888c3be0b593f769a"
I20260806 23:56:10.775599 8586 raft_consensus.cc:2454] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95665a79758b4269b8d3ab90bb11745a in term 1.
I20260806 23:56:10.775709 8399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5a70cd9ae34c1ba5dd09f57ecf9df3" candidate_uuid: "1f42572316994f19b4e7d17941c645a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95665a79758b4269b8d3ab90bb11745a"
I20260806 23:56:10.775843 8399 raft_consensus.cc:2379] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f42572316994f19b4e7d17941c645a8 in current term 1: Already voted for candidate 95665a79758b4269b8d3ab90bb11745a in this term.
I20260806 23:56:10.775924 8298 leader_election.cc:302] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 95665a79758b4269b8d3ab90bb11745a, c46eebf1c2cf44c888c3be0b593f769a; no voters: 1f42572316994f19b4e7d17941c645a8
I20260806 23:56:10.776078 8694 raft_consensus.cc:2781] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:10.776266 8082 leader_election.cc:302] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f42572316994f19b4e7d17941c645a8; no voters: 95665a79758b4269b8d3ab90bb11745a, c46eebf1c2cf44c888c3be0b593f769a
I20260806 23:56:10.776296 8694 raft_consensus.cc:686] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 1 LEADER]: Becoming Leader. State: Replica: 95665a79758b4269b8d3ab90bb11745a, State: Running, Role: LEADER
I20260806 23:56:10.776481 8692 raft_consensus.cc:2726] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 23:56:10.776629 8694 consensus_queue.cc:237] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } }
I20260806 23:56:10.779616 7957 catalog_manager.cc:5495] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a reported cstate change: term changed from 0 to 1, leader changed from <none> to 95665a79758b4269b8d3ab90bb11745a (127.10.225.130). New cstate: current_term: 1 leader_uuid: "95665a79758b4269b8d3ab90bb11745a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95665a79758b4269b8d3ab90bb11745a" member_type: VOTER last_known_addr { host: "127.10.225.130" port: 42999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 } health_report { overall_health: UNKNOWN } } }
I20260806 23:56:10.794601 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.791118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.10.225.130:42999)
0806 23:56:10.791252 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:10.791257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:10.791270 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:56:10.791372 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:10.791374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:10.791430 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:10.791520 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:10.791798 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.792296 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.793154 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:10.793371 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:10.793457 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:10.793759 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:10.794195 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:10.794301 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:10.794406 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 23:56:10.794458 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 23:56:10.797144 8654 tablet.cc:2369] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 23:56:10.800424 8585 raft_consensus.cc:1261] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 1 FOLLOWER]: Refusing update from remote peer 95665a79758b4269b8d3ab90bb11745a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:10.800429 8188 raft_consensus.cc:1261] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8 [term 1 FOLLOWER]: Refusing update from remote peer 95665a79758b4269b8d3ab90bb11745a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:10.800941 8694 consensus_queue.cc:1035] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c46eebf1c2cf44c888c3be0b593f769a" member_type: VOTER last_known_addr { host: "127.10.225.131" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:10.801142 8701 consensus_queue.cc:1035] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f42572316994f19b4e7d17941c645a8" member_type: VOTER last_known_addr { host: "127.10.225.129" port: 35883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:10.808831 8706 mvcc.cc:205] Tried to move back new op lower bound from 7315704097991606272 to 7315704097903443968. Current Snapshot: MvccSnapshot[applied={T|T < 7315704097991606272}]
I20260806 23:56:10.819475 7957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:10.819697 7957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 23:56:10.822280 7957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d21771d979f744079a13b9f7700f8636: Sending DeleteTablet for 3 replicas of tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3
I20260806 23:56:10.823040 8370 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:10 UTC) from {username='slave'} at 127.0.0.1:47346
I20260806 23:56:10.823079 8557 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:10 UTC) from {username='slave'} at 127.0.0.1:33378
I20260806 23:56:10.823238 8158 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 23:56:10 UTC) from {username='slave'} at 127.0.0.1:53838
I20260806 23:56:10.823582 8722 tablet_replica.cc:330] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: stopping tablet replica
I20260806 23:56:10.823740 8723 tablet_replica.cc:330] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: stopping tablet replica
I20260806 23:56:10.823951 8722 raft_consensus.cc:2229] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 23:56:10.824005 8726 tablet_replica.cc:330] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 1f42572316994f19b4e7d17941c645a8: stopping tablet replica
I20260806 23:56:10.824513 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 8057
I20260806 23:56:10.853528 8722 raft_consensus.cc:2258] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:10.854990 8723 raft_consensus.cc:2229] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 1 LEADER]: Raft consensus shutting down.
I20260806 23:56:10.855491 8723 raft_consensus.cc:2258] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 23:56:10.855916 8722 ts_tablet_manager.cc:1905] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:10.857028 8723 ts_tablet_manager.cc:1905] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 23:56:10.858984 8722 ts_tablet_manager.cc:1918] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:10.859180 8722 log.cc:1196] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-2/wal/wals/fb5a70cd9ae34c1ba5dd09f57ecf9df3
I20260806 23:56:10.859611 8722 ts_tablet_manager.cc:1939] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P c46eebf1c2cf44c888c3be0b593f769a: Deleting consensus metadata
I20260806 23:56:10.859733 8723 ts_tablet_manager.cc:1918] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 23:56:10.859833 8723 log.cc:1196] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: Deleting WAL directory at /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786060299689793-11142-0/minicluster-data/ts-1/wal/wals/fb5a70cd9ae34c1ba5dd09f57ecf9df3
I20260806 23:56:10.860122 8723 ts_tablet_manager.cc:1939] T fb5a70cd9ae34c1ba5dd09f57ecf9df3 P 95665a79758b4269b8d3ab90bb11745a: Deleting consensus metadata
I20260806 23:56:10.861313 7936 catalog_manager.cc:4841] TS 95665a79758b4269b8d3ab90bb11745a (127.10.225.130:42999): tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3 (table test-table [id=d2cb8a96f73b46779dc12fbae02594c8]) successfully deleted
I20260806 23:56:10.861315 7939 catalog_manager.cc:4841] TS c46eebf1c2cf44c888c3be0b593f769a (127.10.225.131:38869): tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3 (table test-table [id=d2cb8a96f73b46779dc12fbae02594c8]) successfully deleted
W20260806 23:56:10.861711 7939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.225.129:35883 (error 108)
I20260806 23:56:10.861800 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 8275
I20260806 23:56:10.870790 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:10.870438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.129:35883 (local address 127.0.0.1:53842)
0806 23:56:10.870590 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:10.870681 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.129:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 23:56:10.870995 7939 catalog_manager.cc:4584] TS 1f42572316994f19b4e7d17941c645a8 (127.10.225.129:35883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb5a70cd9ae34c1ba5dd09f57ecf9df3: Network error: Client connection negotiation failed: client connection to 127.10.225.129:35883: connect: Connection refused (error 111)
I20260806 23:56:10.876204 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 8470
I20260806 23:56:10.893081 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 7916
2026-08-06T23:56:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1227 ms)
[----------] 6 tests from EncryptionPolicyTest (11319 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:10 dist-test-slave-nj21 krb5kdc[8734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:10 dist-test-slave-nj21 krb5kdc[8734](info): set up 2 sockets
Aug 06 23:56:10 dist-test-slave-nj21 krb5kdc[8734](info): commencing operation
krb5kdc: starting...
W20260806 23:56:12.029572 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:12 dist-test-slave-nj21 krb5kdc[8734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:12.158594 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:12.272331 8748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:12.272608 8748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:12.272657 8748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:12.276736 8748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:12.276803 8748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:12.276822 8748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:12.276839 8748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:12.276852 8748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:12.284297 8748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:12.285537 8748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:12.286636 8748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:12.295415 8754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:12.295549 8758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:12.295423 8755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:12.295749 8748 server_base.cc:1034] running on GCE node
I20260806 23:56:12.296314 8748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:12.297459 8748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:12.298645 8748 hybrid_clock.cc:648] HybridClock initialized: now 1786060572298630 us; error 27 us; skew 500 ppm
Aug 06 23:56:12 dist-test-slave-nj21 krb5kdc[8734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:12.301448 8748 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:12.302435 8748 webserver.cc:460] Webserver started at http://127.0.0.1:36713/ using document root <none> and password file <none>
I20260806 23:56:12.303122 8748 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:12.303200 8748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:12.303429 8748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:12.305559 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "dfc44bfb60014f04943ea5e9ed19db3b"
format_stamp: "Formatted at 2026-08-06 23:56:12 on dist-test-slave-nj21"
server_key: "4d76a310f60485ce5207f08b764c9caf"
server_key_iv: "5c5bf7eb607ea39868f52d3b5a845cce"
server_key_version: "encryptionkey@0"
I20260806 23:56:12.306104 8748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "dfc44bfb60014f04943ea5e9ed19db3b"
format_stamp: "Formatted at 2026-08-06 23:56:12 on dist-test-slave-nj21"
server_key: "4d76a310f60485ce5207f08b764c9caf"
server_key_iv: "5c5bf7eb607ea39868f52d3b5a845cce"
server_key_version: "encryptionkey@0"
I20260806 23:56:12.309993 8748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:12.312602 8768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:12.313907 8748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 23:56:12.314025 8748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "dfc44bfb60014f04943ea5e9ed19db3b"
format_stamp: "Formatted at 2026-08-06 23:56:12 on dist-test-slave-nj21"
server_key: "4d76a310f60485ce5207f08b764c9caf"
server_key_iv: "5c5bf7eb607ea39868f52d3b5a845cce"
server_key_version: "encryptionkey@0"
I20260806 23:56:12.314136 8748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:12.323731 8748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:12.327229 8748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:12.327409 8748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:12.334904 8748 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38425
I20260806 23:56:12.334923 8853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38425 every 8 connection(s)
I20260806 23:56:12.335948 8748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:12.338806 8854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:12.344681 8854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b: Bootstrap starting.
I20260806 23:56:12.346073 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 8748
I20260806 23:56:12.346220 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:12.346513 11142 external_mini_cluster.cc:1428] Setting key 675c893adc2eafe4782ddaa15c66b685
I20260806 23:56:12.347707 8854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:12.348546 8854 log.cc:824] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:12.351080 8854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b: No bootstrap required, opened a new log
I20260806 23:56:12.355042 8854 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc44bfb60014f04943ea5e9ed19db3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38425 } }
I20260806 23:56:12.355341 8854 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:12.355417 8854 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc44bfb60014f04943ea5e9ed19db3b, State: Initialized, Role: FOLLOWER
Aug 06 23:56:12 dist-test-slave-nj21 krb5kdc[8734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 23:56:12.355901 8854 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc44bfb60014f04943ea5e9ed19db3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38425 } }
I20260806 23:56:12.356045 8854 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:12.356112 8854 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:12.356209 8854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:12.357321 8854 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc44bfb60014f04943ea5e9ed19db3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38425 } }
I20260806 23:56:12.357652 8854 leader_election.cc:302] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfc44bfb60014f04943ea5e9ed19db3b; no voters:
I20260806 23:56:12.357980 8854 leader_election.cc:288] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:12.358065 8862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:12.358247 8862 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [term 1 LEADER]: Becoming Leader. State: Replica: dfc44bfb60014f04943ea5e9ed19db3b, State: Running, Role: LEADER
I20260806 23:56:12.358551 8862 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc44bfb60014f04943ea5e9ed19db3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38425 } }
I20260806 23:56:12.359349 8854 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:12.361267 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:12.347750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41337 (local address 127.0.0.1:38425)
0806 23:56:12.348238 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:12.348253 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:12.348289 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:56:12.349037 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:12.349063 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:12.349578 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:12.349913 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:12.350736 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:12.351802 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:12.352639 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:12.353120 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:12.353268 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:12.356144 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:12.356151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:12.356169 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:12.356203 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:12.357846 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:12.358537 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:12.358543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:12.358549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:12.358707 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:12.359038 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:12.359042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:12.359044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:12.359516 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:12.359741 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:12.360045 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 23:56:12.360280 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":154,"threads_started":1}
I20260806 23:56:12.362339 8864 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfc44bfb60014f04943ea5e9ed19db3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc44bfb60014f04943ea5e9ed19db3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38425 } } }
I20260806 23:56:12.362468 8864 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:12.362723 8865 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfc44bfb60014f04943ea5e9ed19db3b. Latest consensus state: current_term: 1 leader_uuid: "dfc44bfb60014f04943ea5e9ed19db3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc44bfb60014f04943ea5e9ed19db3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38425 } } }
I20260806 23:56:12.362815 8865 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b [sys.catalog]: This master's current role is: LEADER
W20260806 23:56:12.364277 8881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:12.364358 8881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:12.364423 8884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:12.365118 8884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:12.366683 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:12.370839 8884 catalog_manager.cc:1347] Generated new cluster ID: aa2e359e1de24655ba9b7db725fdb7e0
I20260806 23:56:12.370909 8884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:12.377581 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:12.368993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40881 (local address 127.0.0.1:38425)
0806 23:56:12.369100 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:12.369105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:12.369289 (+ 184us) server_negotiation.cc:407] Connection header received
0806 23:56:12.369402 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:12.369405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:12.369450 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:12.369529 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:12.370182 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:12.370844 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:12.371930 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:12.372281 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:12.372346 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:12.374088 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:12.374092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:12.374094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:12.374120 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:12.375658 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:12.376208 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:12.376212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:12.376213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:12.376296 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:12.376613 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:12.376616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:12.376618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:12.376823 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:12.376942 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:12.377312 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 23:56:12.377415 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 23:56:12.429661 8884 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:12.431102 8884 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:12.441537 8884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfc44bfb60014f04943ea5e9ed19db3b: Generated new TSK 0
I20260806 23:56:12.442449 8884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:12.450371 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 8748
W20260806 23:56:12.462839 8729 connection.cc:535] client connection to 127.0.0.1:38425 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38425): Connection reset by peer (error 104)
2026-08-06T23:56:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1558 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:12 dist-test-slave-nj21 krb5kdc[8899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:12 dist-test-slave-nj21 krb5kdc[8899](info): set up 2 sockets
Aug 06 23:56:12 dist-test-slave-nj21 krb5kdc[8899](info): commencing operation
krb5kdc: starting...
W20260806 23:56:13.582533 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:13 dist-test-slave-nj21 krb5kdc[8899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:13.714205 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:13.845831 8918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:13.846133 8918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:13.846194 8918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:13.850318 8918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:13.850394 8918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:13.850423 8918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:13.850445 8918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:13.850466 8918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:13.856068 8918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:13.857214 8918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:13.858317 8918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:13.867214 8923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:13.867214 8928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:13.867437 8926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:13.867627 8918 server_base.cc:1034] running on GCE node
I20260806 23:56:13.868321 8918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:13.869439 8918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:13.870628 8918 hybrid_clock.cc:648] HybridClock initialized: now 1786060573870597 us; error 42 us; skew 500 ppm
Aug 06 23:56:13 dist-test-slave-nj21 krb5kdc[8899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:13.873410 8918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:13.874281 8918 webserver.cc:460] Webserver started at http://127.0.0.1:44721/ using document root <none> and password file <none>
I20260806 23:56:13.874800 8918 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:13.874863 8918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:13.875048 8918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:13.877136 8918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "360ec1d117574f7186ba8190a9b28266"
format_stamp: "Formatted at 2026-08-06 23:56:13 on dist-test-slave-nj21"
server_key: "e3a26d0ef238ac15ecdd4abef44759c1"
server_key_iv: "68fd4ff33151347e59d97327f6fedc9c"
server_key_version: "encryptionkey@0"
I20260806 23:56:13.877655 8918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "360ec1d117574f7186ba8190a9b28266"
format_stamp: "Formatted at 2026-08-06 23:56:13 on dist-test-slave-nj21"
server_key: "e3a26d0ef238ac15ecdd4abef44759c1"
server_key_iv: "68fd4ff33151347e59d97327f6fedc9c"
server_key_version: "encryptionkey@0"
I20260806 23:56:13.881592 8918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:13.884334 8943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:13.885838 8918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:13.885960 8918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "360ec1d117574f7186ba8190a9b28266"
format_stamp: "Formatted at 2026-08-06 23:56:13 on dist-test-slave-nj21"
server_key: "e3a26d0ef238ac15ecdd4abef44759c1"
server_key_iv: "68fd4ff33151347e59d97327f6fedc9c"
server_key_version: "encryptionkey@0"
I20260806 23:56:13.886056 8918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:13.911362 8918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:13.914880 8918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:13.915047 8918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:13.922151 8918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38947
I20260806 23:56:13.922178 9023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38947 every 8 connection(s)
I20260806 23:56:13.923125 8918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:13.923758 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 8918
I20260806 23:56:13.923908 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:13.924178 11142 external_mini_cluster.cc:1428] Setting key c9884724d812863fc6f76094de6d73eb
I20260806 23:56:13.926352 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:13.932070 9025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266: Bootstrap starting.
I20260806 23:56:13.934754 9025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:13.934790 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:13.925880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39857 (local address 127.0.0.1:38947)
0806 23:56:13.926353 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:13.926368 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:13.926407 (+ 39us) server_negotiation.cc:407] Connection header received
0806 23:56:13.927088 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:13.927104 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:13.927787 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:13.928394 (+ 607us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:13.929042 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:13.930080 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:13.930989 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:13.931460 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:13.931613 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:13.932588 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:13.932593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:13.932606 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:13.933203 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:13.933302 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:13.933341 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:13.933612 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:13.933805 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 23:56:13.933966 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":150,"threads_started":1}
I20260806 23:56:13.935976 9025 log.cc:824] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:13.938089 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:13.938179 9025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266: No bootstrap required, opened a new log
I20260806 23:56:13.942219 9025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360ec1d117574f7186ba8190a9b28266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38947 } }
I20260806 23:56:13.942471 9025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:13.942507 9025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360ec1d117574f7186ba8190a9b28266, State: Initialized, Role: FOLLOWER
I20260806 23:56:13.942941 9025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360ec1d117574f7186ba8190a9b28266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38947 } }
I20260806 23:56:13.943081 9025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:13.943148 9025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:13.943260 9025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:13.944293 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:13.940118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56791 (local address 127.0.0.1:38947)
0806 23:56:13.940252 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:13.940257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:13.940471 (+ 214us) server_negotiation.cc:407] Connection header received
0806 23:56:13.940584 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:13.940588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:13.940633 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:13.940703 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:13.940956 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:13.941572 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:13.942645 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:13.942887 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:13.942976 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:13.943835 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:13.943839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:13.943842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:13.943856 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:13.943892 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:13.943903 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:13.943976 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:13.944117 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 23:56:13.944156 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 23:56:13.944406 9025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360ec1d117574f7186ba8190a9b28266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38947 } }
I20260806 23:56:13.944787 9025 leader_election.cc:302] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 360ec1d117574f7186ba8190a9b28266; no voters:
I20260806 23:56:13.945094 9025 leader_election.cc:288] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:13.945223 9038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:13.945432 9038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [term 1 LEADER]: Becoming Leader. State: Replica: 360ec1d117574f7186ba8190a9b28266, State: Running, Role: LEADER
I20260806 23:56:13.945744 9038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360ec1d117574f7186ba8190a9b28266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38947 } }
I20260806 23:56:13.946290 9025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:13.947299 9043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 360ec1d117574f7186ba8190a9b28266. Latest consensus state: current_term: 1 leader_uuid: "360ec1d117574f7186ba8190a9b28266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360ec1d117574f7186ba8190a9b28266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38947 } } }
I20260806 23:56:13.947427 9043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:13.947753 9039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "360ec1d117574f7186ba8190a9b28266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360ec1d117574f7186ba8190a9b28266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38947 } } }
I20260806 23:56:13.947862 9039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266 [sys.catalog]: This master's current role is: LEADER
W20260806 23:56:13.950697 9067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:13.950799 9067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:13.951056 9057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:13.951814 9057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:13.957456 9057 catalog_manager.cc:1347] Generated new cluster ID: 69dd3394cfc348e69a342237eda9dc22
I20260806 23:56:13.957532 9057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:13.966668 9057 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:13.967705 9057 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:13.976552 9057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 360ec1d117574f7186ba8190a9b28266: Generated new TSK 0
I20260806 23:56:13.977316 9057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:13.984455 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 8918
W20260806 23:56:13.996594 8896 connection.cc:535] client connection to 127.0.0.1:38947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38947): Connection reset by peer (error 104)
2026-08-06T23:56:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1533 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:14 dist-test-slave-nj21 krb5kdc[9078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:14 dist-test-slave-nj21 krb5kdc[9078](info): set up 2 sockets
Aug 06 23:56:14 dist-test-slave-nj21 krb5kdc[9078](info): commencing operation
krb5kdc: starting...
W20260806 23:56:15.118854 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:15 dist-test-slave-nj21 krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:15.247035 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:15.367194 9095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:15.367504 9095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:15.367568 9095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:15.371801 9095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:15.371879 9095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:15.371906 9095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:15.371930 9095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:15.371959 9095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:15.379091 9095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:15.380997 9095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:15.382304 9095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:15.391127 9106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:15.391127 9104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:15.391127 9108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:15.391500 9095 server_base.cc:1034] running on GCE node
I20260806 23:56:15.392328 9095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:15.393491 9095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:15.394737 9095 hybrid_clock.cc:648] HybridClock initialized: now 1786060575394702 us; error 28 us; skew 500 ppm
Aug 06 23:56:15 dist-test-slave-nj21 krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:15.406262 9095 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:15.407467 9095 webserver.cc:460] Webserver started at http://127.0.0.1:33957/ using document root <none> and password file <none>
I20260806 23:56:15.408106 9095 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:15.408172 9095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:15.408378 9095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:15.410516 9095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "76cde2132d254733b789742dccb166eb"
format_stamp: "Formatted at 2026-08-06 23:56:15 on dist-test-slave-nj21"
server_key: "0c08dc69286bb9e8ff626522fa21162e"
server_key_iv: "3541c896abd37b3a863f19a93dab812f"
server_key_version: "encryptionkey@0"
I20260806 23:56:15.411082 9095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "76cde2132d254733b789742dccb166eb"
format_stamp: "Formatted at 2026-08-06 23:56:15 on dist-test-slave-nj21"
server_key: "0c08dc69286bb9e8ff626522fa21162e"
server_key_iv: "3541c896abd37b3a863f19a93dab812f"
server_key_version: "encryptionkey@0"
I20260806 23:56:15.415295 9095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 23:56:15.418120 9117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:15.419513 9095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 23:56:15.419629 9095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "76cde2132d254733b789742dccb166eb"
format_stamp: "Formatted at 2026-08-06 23:56:15 on dist-test-slave-nj21"
server_key: "0c08dc69286bb9e8ff626522fa21162e"
server_key_iv: "3541c896abd37b3a863f19a93dab812f"
server_key_version: "encryptionkey@0"
I20260806 23:56:15.419775 9095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:15.424444 9095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:15.428107 9095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:15.428272 9095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:15.437021 9095 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34265
I20260806 23:56:15.437072 9194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34265 every 8 connection(s)
I20260806 23:56:15.438102 9095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:15.441330 9195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:15.444700 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 9095
I20260806 23:56:15.444854 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:15.445171 11142 external_mini_cluster.cc:1428] Setting key 2622f643024193c2d5484f08d00b3c04
I20260806 23:56:15.448174 9195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb: Bootstrap starting.
I20260806 23:56:15.451392 9195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:15.451412 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:15.447028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33963 (local address 127.0.0.1:34265)
0806 23:56:15.447651 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:15.447666 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:15.447744 (+ 78us) server_negotiation.cc:407] Connection header received
0806 23:56:15.448579 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:15.448602 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:15.448922 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:15.449257 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:15.450181 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:15.450186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:15.450199 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:15.450211 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:15.450268 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:15.450290 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:15.450432 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:15.450503 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 23:56:15.450662 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":249,"threads_started":1}
I20260806 23:56:15.452234 9195 log.cc:824] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:15.454564 9195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb: No bootstrap required, opened a new log
I20260806 23:56:15.454607 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:15.458626 9195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cde2132d254733b789742dccb166eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260806 23:56:15.458889 9195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:15.458948 9195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76cde2132d254733b789742dccb166eb, State: Initialized, Role: FOLLOWER
I20260806 23:56:15.459406 9195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cde2132d254733b789742dccb166eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260806 23:56:15.459429 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:15.457451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37025 (local address 127.0.0.1:34265)
0806 23:56:15.457570 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:15.457574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:15.457647 (+ 73us) server_negotiation.cc:407] Connection header received
0806 23:56:15.457778 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:15.457781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:15.457820 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:15.457884 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:15.458781 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:15.458785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:15.458787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:15.458798 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:15.458826 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:15.458837 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:15.458910 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:15.459148 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 23:56:15.459190 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:56:15.459539 9195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:15.459591 9195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:15.459707 9195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:15.460934 9195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cde2132d254733b789742dccb166eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260806 23:56:15.461247 9195 leader_election.cc:302] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76cde2132d254733b789742dccb166eb; no voters:
I20260806 23:56:15.461512 9195 leader_election.cc:288] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:15.461642 9212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:15.461867 9212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [term 1 LEADER]: Becoming Leader. State: Replica: 76cde2132d254733b789742dccb166eb, State: Running, Role: LEADER
I20260806 23:56:15.462159 9212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cde2132d254733b789742dccb166eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260806 23:56:15.462741 9195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:15.464250 9213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76cde2132d254733b789742dccb166eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cde2132d254733b789742dccb166eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } } }
I20260806 23:56:15.464247 9214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76cde2132d254733b789742dccb166eb. Latest consensus state: current_term: 1 leader_uuid: "76cde2132d254733b789742dccb166eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cde2132d254733b789742dccb166eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } } }
I20260806 23:56:15.464748 9214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:15.464748 9213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:15.465512 9239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:15.469038 9239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:15.475347 9239 catalog_manager.cc:1347] Generated new cluster ID: b0f6f94b947c4696ae6364ea1f2f69e1
I20260806 23:56:15.475440 9239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:15.490507 9239 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:15.490880 9239 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:15.499927 9239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76cde2132d254733b789742dccb166eb: Generated new TSK 0
I20260806 23:56:15.500857 9239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:15.508975 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 9095
2026-08-06T23:56:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1524 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:15 dist-test-slave-nj21 krb5kdc[9247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:15 dist-test-slave-nj21 krb5kdc[9247](info): set up 2 sockets
Aug 06 23:56:15 dist-test-slave-nj21 krb5kdc[9247](info): commencing operation
krb5kdc: starting...
W20260806 23:56:16.681139 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:16 dist-test-slave-nj21 krb5kdc[9247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:16.819136 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:16.939558 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:16.939918 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:16.939983 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:16.944569 9267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:16.944655 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:16.944687 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:16.944708 9267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:16.944731 9267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:16.950408 9267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:16.951640 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:16.952730 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:16.961226 9278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:16.961230 9276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:16.961366 9267 server_base.cc:1034] running on GCE node
W20260806 23:56:16.961246 9286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:16.962090 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:16.963272 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:16.964445 9267 hybrid_clock.cc:648] HybridClock initialized: now 1786060576964429 us; error 34 us; skew 500 ppm
Aug 06 23:56:16 dist-test-slave-nj21 krb5kdc[9247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:16.967331 9267 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:16.968317 9267 webserver.cc:460] Webserver started at http://127.0.0.1:34015/ using document root <none> and password file <none>
I20260806 23:56:16.968875 9267 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:16.968936 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:16.969105 9267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:16.971212 9267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "616ab7b8b2fe41b5ada94777c2e75cd9"
format_stamp: "Formatted at 2026-08-06 23:56:16 on dist-test-slave-nj21"
server_key: "212af4c366ad59dac970ca4f8b8801e9"
server_key_iv: "5103e56cc7d4e4c32a99a8619d8c8f85"
server_key_version: "encryptionkey@0"
I20260806 23:56:16.971791 9267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "616ab7b8b2fe41b5ada94777c2e75cd9"
format_stamp: "Formatted at 2026-08-06 23:56:16 on dist-test-slave-nj21"
server_key: "212af4c366ad59dac970ca4f8b8801e9"
server_key_iv: "5103e56cc7d4e4c32a99a8619d8c8f85"
server_key_version: "encryptionkey@0"
I20260806 23:56:16.976850 9267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 23:56:16.979610 9293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:16.981235 9267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:16.981376 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "616ab7b8b2fe41b5ada94777c2e75cd9"
format_stamp: "Formatted at 2026-08-06 23:56:16 on dist-test-slave-nj21"
server_key: "212af4c366ad59dac970ca4f8b8801e9"
server_key_iv: "5103e56cc7d4e4c32a99a8619d8c8f85"
server_key_version: "encryptionkey@0"
I20260806 23:56:16.981488 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:16.996965 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:17.000442 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:17.000613 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:17.007694 9267 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34099
I20260806 23:56:17.007742 9368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34099 every 8 connection(s)
I20260806 23:56:17.008687 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:17.011564 9369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:17.017405 9369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9: Bootstrap starting.
I20260806 23:56:17.018041 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 9267
I20260806 23:56:17.018177 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:17.018435 11142 external_mini_cluster.cc:1428] Setting key 0b00dee94c8773f0e35ae065a1a22bc3
I20260806 23:56:17.020321 9369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:17.021235 9369 log.cc:824] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:17.023223 9369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9: No bootstrap required, opened a new log
I20260806 23:56:17.026887 9369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ab7b8b2fe41b5ada94777c2e75cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34099 } }
I20260806 23:56:17.027207 9369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:17.027263 9369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616ab7b8b2fe41b5ada94777c2e75cd9, State: Initialized, Role: FOLLOWER
Aug 06 23:56:17 dist-test-slave-nj21 krb5kdc[9247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 23:56:17.027863 9369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ab7b8b2fe41b5ada94777c2e75cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34099 } }
I20260806 23:56:17.028041 9369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:17.028120 9369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:17.028225 9369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:17.029316 9369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ab7b8b2fe41b5ada94777c2e75cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34099 } }
I20260806 23:56:17.029716 9369 leader_election.cc:302] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 616ab7b8b2fe41b5ada94777c2e75cd9; no voters:
I20260806 23:56:17.030109 9369 leader_election.cc:288] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:17.030225 9377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:17.030467 9377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [term 1 LEADER]: Becoming Leader. State: Replica: 616ab7b8b2fe41b5ada94777c2e75cd9, State: Running, Role: LEADER
I20260806 23:56:17.030814 9377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ab7b8b2fe41b5ada94777c2e75cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34099 } }
I20260806 23:56:17.031527 9369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:17.032617 9381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "616ab7b8b2fe41b5ada94777c2e75cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ab7b8b2fe41b5ada94777c2e75cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34099 } } }
I20260806 23:56:17.032745 9381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:17.033037 9378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 616ab7b8b2fe41b5ada94777c2e75cd9. Latest consensus state: current_term: 1 leader_uuid: "616ab7b8b2fe41b5ada94777c2e75cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616ab7b8b2fe41b5ada94777c2e75cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34099 } } }
I20260806 23:56:17.033300 9378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:17.034480 9393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:17.034796 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:17.019845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43197 (local address 127.0.0.1:34099)
0806 23:56:17.020340 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:17.020366 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:17.020410 (+ 44us) server_negotiation.cc:407] Connection header received
0806 23:56:17.021271 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:17.021294 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:17.021734 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:17.022046 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:17.022848 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:17.023884 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:17.024709 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:17.025177 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:17.025336 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:17.028225 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:17.028231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:17.028245 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:17.028275 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:17.030742 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:17.031277 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:17.031283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:17.031289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:17.031440 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:17.031761 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:17.031765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:17.031767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:17.032340 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:17.032543 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:17.033429 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 23:56:17.033765 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":134,"threads_started":1}
I20260806 23:56:17.037576 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:17.037952 9393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:17.045976 9393 catalog_manager.cc:1347] Generated new cluster ID: 6e3f2972c5e1491cae91cd2fba4ea274
I20260806 23:56:17.046144 9393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:17.051265 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:17.039664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48489 (local address 127.0.0.1:34099)
0806 23:56:17.039833 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:17.039838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:17.040687 (+ 849us) server_negotiation.cc:407] Connection header received
0806 23:56:17.040987 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:17.040990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:17.041051 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:17.041128 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:17.042173 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:17.043007 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:17.044212 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:17.044465 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:17.044577 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:17.046412 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:17.046417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:17.046420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:17.046457 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:17.049418 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:17.049999 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:17.050002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:17.050004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:17.050100 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:17.050341 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:17.050345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:17.050347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:17.050582 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:17.050731 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:17.050944 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 23:56:17.051079 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 23:56:17.082471 9393 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:17.083592 9393 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:17.092594 9393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 616ab7b8b2fe41b5ada94777c2e75cd9: Generated new TSK 0
I20260806 23:56:17.093343 9393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:17.101471 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 9267
2026-08-06T23:56:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1594 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:17 dist-test-slave-nj21 krb5kdc[9416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:17 dist-test-slave-nj21 krb5kdc[9416](info): set up 2 sockets
Aug 06 23:56:17 dist-test-slave-nj21 krb5kdc[9416](info): commencing operation
krb5kdc: starting...
W20260806 23:56:18.248085 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:18 dist-test-slave-nj21 krb5kdc[9416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:18.384775 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:18.502920 9435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:18.503418 9435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:18.503629 9435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:18.508740 9435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:18.508836 9435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:18.508858 9435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:18.508879 9435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:18.508904 9435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:18.515864 9435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:18.517064 9435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:18.518131 9435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:18.526738 9445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:18.526773 9447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:18.526738 9444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:18.526845 9435 server_base.cc:1034] running on GCE node
I20260806 23:56:18.528048 9435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:18.530048 9435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:18.531296 9435 hybrid_clock.cc:648] HybridClock initialized: now 1786060578531233 us; error 78 us; skew 500 ppm
Aug 06 23:56:18 dist-test-slave-nj21 krb5kdc[9416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:18.534335 9435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:18.535456 9435 webserver.cc:460] Webserver started at http://127.0.0.1:46033/ using document root <none> and password file <none>
I20260806 23:56:18.536193 9435 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:18.536284 9435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:18.536540 9435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:18.539059 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "751b24c01fe24011b801a8c380a32295"
format_stamp: "Formatted at 2026-08-06 23:56:18 on dist-test-slave-nj21"
server_key: "60b202939ef9dbaf3cc4c2935a7470e0"
server_key_iv: "9c8bef7d21b2447266bb2846f24af464"
server_key_version: "encryptionkey@0"
I20260806 23:56:18.539600 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "751b24c01fe24011b801a8c380a32295"
format_stamp: "Formatted at 2026-08-06 23:56:18 on dist-test-slave-nj21"
server_key: "60b202939ef9dbaf3cc4c2935a7470e0"
server_key_iv: "9c8bef7d21b2447266bb2846f24af464"
server_key_version: "encryptionkey@0"
I20260806 23:56:18.543453 9435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:56:18.546792 9457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:18.548398 9435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:18.548544 9435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "751b24c01fe24011b801a8c380a32295"
format_stamp: "Formatted at 2026-08-06 23:56:18 on dist-test-slave-nj21"
server_key: "60b202939ef9dbaf3cc4c2935a7470e0"
server_key_iv: "9c8bef7d21b2447266bb2846f24af464"
server_key_version: "encryptionkey@0"
I20260806 23:56:18.548655 9435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:18.567314 9435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:18.571919 9435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:18.572147 9435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:18.580295 9435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39317
I20260806 23:56:18.580375 9545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39317 every 8 connection(s)
I20260806 23:56:18.581305 9435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:18.582882 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 9435
I20260806 23:56:18.583020 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:18.583273 11142 external_mini_cluster.cc:1428] Setting key 4a9828b9b4d3f18516eee8b9705e5aca
I20260806 23:56:18.584371 9548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:18.590590 9548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295: Bootstrap starting.
I20260806 23:56:18.592108 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:18.584711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60633 (local address 127.0.0.1:39317)
0806 23:56:18.585178 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:18.585193 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:18.585229 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:56:18.585888 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:18.585914 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:18.586241 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:18.586549 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:18.586953 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:18.588020 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:18.588901 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:18.589374 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:18.589541 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:18.590362 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:18.590368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:18.590381 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:18.590801 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:18.590866 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:18.590890 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:18.590977 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:18.591107 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 23:56:18.591332 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":148,"threads_started":1}
I20260806 23:56:18.593391 9548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:18.594245 9548 log.cc:824] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:18.595126 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:18.596560 9548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295: No bootstrap required, opened a new log
I20260806 23:56:18.601114 9548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751b24c01fe24011b801a8c380a32295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } }
I20260806 23:56:18.601680 9548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:18.601802 9548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751b24c01fe24011b801a8c380a32295, State: Initialized, Role: FOLLOWER
I20260806 23:56:18.602020 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:18.597154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58043 (local address 127.0.0.1:39317)
0806 23:56:18.597281 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:18.597286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:18.597414 (+ 128us) server_negotiation.cc:407] Connection header received
0806 23:56:18.597510 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:18.597513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:18.597565 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:18.597662 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:18.597935 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:18.598691 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:18.599913 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:18.600281 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:18.600411 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:18.601358 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:18.601378 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:18.601383 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:18.601414 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:18.601470 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:18.601533 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:18.601654 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:18.601823 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 23:56:18.601889 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 23:56:18.602699 9548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751b24c01fe24011b801a8c380a32295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } }
I20260806 23:56:18.602910 9548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:18.602998 9548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:18.603102 9548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:18.604183 9548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751b24c01fe24011b801a8c380a32295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } }
I20260806 23:56:18.604524 9548 leader_election.cc:302] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 751b24c01fe24011b801a8c380a32295; no voters:
I20260806 23:56:18.604843 9548 leader_election.cc:288] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:18.604992 9564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:18.605195 9564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [term 1 LEADER]: Becoming Leader. State: Replica: 751b24c01fe24011b801a8c380a32295, State: Running, Role: LEADER
I20260806 23:56:18.605522 9564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751b24c01fe24011b801a8c380a32295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } }
I20260806 23:56:18.606137 9548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:18.607347 9567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751b24c01fe24011b801a8c380a32295. Latest consensus state: current_term: 1 leader_uuid: "751b24c01fe24011b801a8c380a32295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751b24c01fe24011b801a8c380a32295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } } }
I20260806 23:56:18.607497 9567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:18.607887 9575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:18.607978 9566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "751b24c01fe24011b801a8c380a32295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751b24c01fe24011b801a8c380a32295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39317 } } }
I20260806 23:56:18.608140 9566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:18.611032 9575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:18.616914 9575 catalog_manager.cc:1347] Generated new cluster ID: b646bbdb19ad4058807f476dcae9d6cc
I20260806 23:56:18.616989 9575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:18.627918 9575 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:18.628871 9575 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:18.634073 9575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 751b24c01fe24011b801a8c380a32295: Generated new TSK 0
I20260806 23:56:18.635039 9575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:18.647833 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 9435
2026-08-06T23:56:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1543 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:18 dist-test-slave-nj21 krb5kdc[9596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:18 dist-test-slave-nj21 krb5kdc[9596](info): set up 2 sockets
Aug 06 23:56:18 dist-test-slave-nj21 krb5kdc[9596](info): commencing operation
krb5kdc: starting...
W20260806 23:56:19.808466 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:19 dist-test-slave-nj21 krb5kdc[9596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:19.944850 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:20.068605 9615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:20.068949 9615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:20.069022 9615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:20.073271 9615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:20.073351 9615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:20.073385 9615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:20.073421 9615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:20.073457 9615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:20.079100 9615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:20.080296 9615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:20.081377 9615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:20.090323 9638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:20.090341 9615 server_base.cc:1034] running on GCE node
W20260806 23:56:20.090241 9625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:20.090329 9633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:20.091187 9615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:20.092923 9615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:20.094107 9615 hybrid_clock.cc:648] HybridClock initialized: now 1786060580094080 us; error 64 us; skew 500 ppm
Aug 06 23:56:20 dist-test-slave-nj21 krb5kdc[9596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:20.097023 9615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:20.097956 9615 webserver.cc:460] Webserver started at http://127.0.0.1:34735/ using document root <none> and password file <none>
I20260806 23:56:20.098863 9615 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:20.098976 9615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:20.099231 9615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:20.101425 9615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "f1bc2eb9286d4c20a53d5800a5111f44"
format_stamp: "Formatted at 2026-08-06 23:56:20 on dist-test-slave-nj21"
server_key: "552fb26b71c72073f263b28619a30748"
server_key_iv: "65f2c2eb6b44643beb9c6e9fc8ba103d"
server_key_version: "encryptionkey@0"
I20260806 23:56:20.101996 9615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "f1bc2eb9286d4c20a53d5800a5111f44"
format_stamp: "Formatted at 2026-08-06 23:56:20 on dist-test-slave-nj21"
server_key: "552fb26b71c72073f263b28619a30748"
server_key_iv: "65f2c2eb6b44643beb9c6e9fc8ba103d"
server_key_version: "encryptionkey@0"
I20260806 23:56:20.105804 9615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:20.108572 9658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:20.109999 9615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:20.110147 9615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "f1bc2eb9286d4c20a53d5800a5111f44"
format_stamp: "Formatted at 2026-08-06 23:56:20 on dist-test-slave-nj21"
server_key: "552fb26b71c72073f263b28619a30748"
server_key_iv: "65f2c2eb6b44643beb9c6e9fc8ba103d"
server_key_version: "encryptionkey@0"
I20260806 23:56:20.110261 9615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:20.115116 9615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:20.118259 9615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:20.118536 9615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:20.126097 9615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44059
I20260806 23:56:20.126171 9737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44059 every 8 connection(s)
I20260806 23:56:20.127121 9615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:20.130398 9739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:20.132947 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 9615
I20260806 23:56:20.133091 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:20.133379 11142 external_mini_cluster.cc:1428] Setting key 7f0598415bed0a59d84998ac33892d62
I20260806 23:56:20.136584 9739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44: Bootstrap starting.
I20260806 23:56:20.138748 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:20.134879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41181 (local address 127.0.0.1:44059)
0806 23:56:20.135340 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:20.135360 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:20.135399 (+ 39us) server_negotiation.cc:407] Connection header received
0806 23:56:20.136091 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:20.136110 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:20.136409 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:20.136706 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:20.137502 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:20.137507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:20.137519 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:20.137531 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:20.137588 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:20.137609 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:20.137721 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:20.137831 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 23:56:20.137983 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":151,"threads_started":1}
I20260806 23:56:20.139103 9739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:20.139948 9739 log.cc:824] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:20.141733 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:20.142109 9739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44: No bootstrap required, opened a new log
I20260806 23:56:20.145812 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:20.144018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56513 (local address 127.0.0.1:44059)
0806 23:56:20.144191 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:20.144195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:20.144319 (+ 124us) server_negotiation.cc:407] Connection header received
0806 23:56:20.144462 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:20.144465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:20.144504 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:20.144602 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:20.145427 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:20.145431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:20.145433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:20.145448 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:20.145483 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:20.145497 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:20.145596 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:20.145631 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 23:56:20.145667 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 23:56:20.146505 9739 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bc2eb9286d4c20a53d5800a5111f44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } }
I20260806 23:56:20.146750 9739 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:20.146809 9739 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1bc2eb9286d4c20a53d5800a5111f44, State: Initialized, Role: FOLLOWER
I20260806 23:56:20.147246 9739 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bc2eb9286d4c20a53d5800a5111f44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } }
I20260806 23:56:20.147397 9739 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:20.147445 9739 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:20.147536 9739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:20.148586 9739 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bc2eb9286d4c20a53d5800a5111f44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } }
I20260806 23:56:20.148929 9739 leader_election.cc:302] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1bc2eb9286d4c20a53d5800a5111f44; no voters:
I20260806 23:56:20.149238 9739 leader_election.cc:288] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:20.149334 9750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:20.149613 9750 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [term 1 LEADER]: Becoming Leader. State: Replica: f1bc2eb9286d4c20a53d5800a5111f44, State: Running, Role: LEADER
I20260806 23:56:20.149924 9750 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bc2eb9286d4c20a53d5800a5111f44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } }
I20260806 23:56:20.150540 9739 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:20.151556 9752 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1bc2eb9286d4c20a53d5800a5111f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bc2eb9286d4c20a53d5800a5111f44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } } }
I20260806 23:56:20.151729 9752 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:20.152325 9763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:20.152603 9753 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1bc2eb9286d4c20a53d5800a5111f44. Latest consensus state: current_term: 1 leader_uuid: "f1bc2eb9286d4c20a53d5800a5111f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1bc2eb9286d4c20a53d5800a5111f44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44059 } } }
I20260806 23:56:20.152720 9753 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:20.156095 9763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:20.162967 9763 catalog_manager.cc:1347] Generated new cluster ID: 0e8f415413eb400c947fe673297fda73
I20260806 23:56:20.163059 9763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:20.175330 9763 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:20.175729 9763 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:20.180991 9763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1bc2eb9286d4c20a53d5800a5111f44: Generated new TSK 0
I20260806 23:56:20.181687 9763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:20.186609 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 9615
2026-08-06T23:56:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1541 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:20 dist-test-slave-nj21 krb5kdc[9785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:20 dist-test-slave-nj21 krb5kdc[9785](info): set up 2 sockets
Aug 06 23:56:20 dist-test-slave-nj21 krb5kdc[9785](info): commencing operation
krb5kdc: starting...
W20260806 23:56:21.330508 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:21 dist-test-slave-nj21 krb5kdc[9785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:21.466850 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:21.584162 9804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:21.584645 9804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:21.584699 9804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:21.590574 9804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:21.590682 9804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:21.590703 9804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:21.590721 9804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:21.590736 9804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:21.597271 9804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:21.598392 9804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:21.599386 9804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:21.607795 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:21.607821 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:21.607985 9804 server_base.cc:1034] running on GCE node
W20260806 23:56:21.607800 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:21.608750 9804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:21.609880 9804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:21.611050 9804 hybrid_clock.cc:648] HybridClock initialized: now 1786060581611036 us; error 31 us; skew 500 ppm
Aug 06 23:56:21 dist-test-slave-nj21 krb5kdc[9785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:21.618605 9804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:21.619756 9804 webserver.cc:460] Webserver started at http://127.0.0.1:44283/ using document root <none> and password file <none>
I20260806 23:56:21.620366 9804 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:21.620435 9804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:21.620640 9804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:21.623001 9804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "ded762382a3744e08560205c9eace62e"
format_stamp: "Formatted at 2026-08-06 23:56:21 on dist-test-slave-nj21"
server_key: "7358070a3e3f6d082b9ad2a52e3a0abe"
server_key_iv: "43463dca3695a42537a82f543f477a31"
server_key_version: "encryptionkey@0"
I20260806 23:56:21.623624 9804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "ded762382a3744e08560205c9eace62e"
format_stamp: "Formatted at 2026-08-06 23:56:21 on dist-test-slave-nj21"
server_key: "7358070a3e3f6d082b9ad2a52e3a0abe"
server_key_iv: "43463dca3695a42537a82f543f477a31"
server_key_version: "encryptionkey@0"
I20260806 23:56:21.627895 9804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:56:21.630504 9828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:21.632527 9804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:21.632675 9804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "ded762382a3744e08560205c9eace62e"
format_stamp: "Formatted at 2026-08-06 23:56:21 on dist-test-slave-nj21"
server_key: "7358070a3e3f6d082b9ad2a52e3a0abe"
server_key_iv: "43463dca3695a42537a82f543f477a31"
server_key_version: "encryptionkey@0"
I20260806 23:56:21.632790 9804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:21.664242 9804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:21.667393 9804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:21.667593 9804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:21.674491 9804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34295
I20260806 23:56:21.674523 9903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34295 every 8 connection(s)
I20260806 23:56:21.675740 9804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:21.679690 9904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:21.684780 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 9804
I20260806 23:56:21.684942 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:21.685243 11142 external_mini_cluster.cc:1428] Setting key 59722d201415472201b0f88f04102094
I20260806 23:56:21.686224 9904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e: Bootstrap starting.
I20260806 23:56:21.688759 9904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:21.689702 9904 log.cc:824] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:21.691866 9904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e: No bootstrap required, opened a new log
Aug 06 23:56:21 dist-test-slave-nj21 krb5kdc[9785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 23:56:21.697973 9904 raft_consensus.cc:348] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded762382a3744e08560205c9eace62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } }
I20260806 23:56:21.698248 9904 raft_consensus.cc:374] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:21.698323 9904 raft_consensus.cc:729] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded762382a3744e08560205c9eace62e, State: Initialized, Role: FOLLOWER
I20260806 23:56:21.698797 9904 consensus_queue.cc:260] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded762382a3744e08560205c9eace62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } }
I20260806 23:56:21.698942 9904 raft_consensus.cc:388] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:21.699010 9904 raft_consensus.cc:482] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:21.699111 9904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:21.700155 9904 raft_consensus.cc:504] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded762382a3744e08560205c9eace62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } }
I20260806 23:56:21.700539 9904 leader_election.cc:302] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ded762382a3744e08560205c9eace62e; no voters:
I20260806 23:56:21.700884 9904 leader_election.cc:288] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:21.701005 9910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:21.701220 9910 raft_consensus.cc:686] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [term 1 LEADER]: Becoming Leader. State: Replica: ded762382a3744e08560205c9eace62e, State: Running, Role: LEADER
I20260806 23:56:21.701547 9910 consensus_queue.cc:237] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded762382a3744e08560205c9eace62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } }
I20260806 23:56:21.701606 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:21.687191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:39127 (local address 127.0.0.1:34295)
0806 23:56:21.687625 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:21.687642 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:21.687711 (+ 69us) server_negotiation.cc:407] Connection header received
0806 23:56:21.688391 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:21.688407 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:21.688749 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:21.689064 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:21.690008 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:21.691003 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:21.692136 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:21.692623 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:21.692758 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:21.696504 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:21.696510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:21.696526 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:21.696558 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:21.698188 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:21.698800 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:21.698805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:21.698810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:21.698932 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:21.699303 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:21.699306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:21.699308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:21.699794 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:21.699995 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:21.700333 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 23:56:21.700575 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":131,"threads_started":1}
I20260806 23:56:21.702229 9904 sys_catalog.cc:564] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:21.702870 9910 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ded762382a3744e08560205c9eace62e. Latest consensus state: current_term: 1 leader_uuid: "ded762382a3744e08560205c9eace62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded762382a3744e08560205c9eace62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } } }
I20260806 23:56:21.702998 9910 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:21.703298 9911 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ded762382a3744e08560205c9eace62e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded762382a3744e08560205c9eace62e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } } }
I20260806 23:56:21.703414 9911 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:21.703941 9920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:21.705284 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:21.707521 9920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:21.714916 9920 catalog_manager.cc:1347] Generated new cluster ID: 928f2b8a66de45aebb32dff88de44c98
I20260806 23:56:21.714995 9920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:21.716305 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:21.707368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:42117 (local address 127.0.0.1:34295)
0806 23:56:21.707496 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:21.707501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:21.707813 (+ 312us) server_negotiation.cc:407] Connection header received
0806 23:56:21.708000 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:21.708003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:21.708052 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:21.708094 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:21.708838 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:21.709343 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:21.710126 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:21.710357 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:21.710443 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:21.711784 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:21.711787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:21.711790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:21.711813 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:21.713260 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:21.713829 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:21.713832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:21.713834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:21.713927 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:21.714281 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:21.714285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:21.714286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:21.714519 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:21.714655 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:21.716052 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0806 23:56:21.716152 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:56:21.741362 9920 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:21.742437 9920 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:21.752056 9920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ded762382a3744e08560205c9eace62e: Generated new TSK 0
I20260806 23:56:21.752743 9920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:21.764602 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 9804
W20260806 23:56:21.779548 9783 connection.cc:535] client connection to 127.0.0.1:34295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34295): Connection reset by peer (error 104)
2026-08-06T23:56:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:21 dist-test-slave-nj21 krb5kdc[9946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:21 dist-test-slave-nj21 krb5kdc[9946](info): set up 2 sockets
Aug 06 23:56:21 dist-test-slave-nj21 krb5kdc[9946](info): commencing operation
krb5kdc: starting...
W20260806 23:56:22.910722 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:22 dist-test-slave-nj21 krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:23.040712 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:23.155362 9962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:23.155663 9962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:23.155762 9962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:23.159889 9962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:23.159955 9962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:23.159982 9962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:23.160005 9962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:23.160033 9962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:23.165436 9962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:23.166563 9962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:23.167585 9962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:23.176013 9972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:23.176003 9977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:23.176095 9962 server_base.cc:1034] running on GCE node
W20260806 23:56:23.176005 9973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:23.177083 9962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:23.178252 9962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:23.179440 9962 hybrid_clock.cc:648] HybridClock initialized: now 1786060583179426 us; error 27 us; skew 500 ppm
Aug 06 23:56:23 dist-test-slave-nj21 krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:23.182788 9962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:23.184417 9962 webserver.cc:460] Webserver started at http://127.0.0.1:45703/ using document root <none> and password file <none>
I20260806 23:56:23.185151 9962 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:23.185243 9962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:23.185622 9962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:23.188035 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "42fcbedce7794d5588b5b8966587f46b"
format_stamp: "Formatted at 2026-08-06 23:56:23 on dist-test-slave-nj21"
server_key: "014548423073ff81b14b4ac926dbb6b5"
server_key_iv: "8645e17c8165c22d28168e3a324467c3"
server_key_version: "encryptionkey@0"
I20260806 23:56:23.188596 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "42fcbedce7794d5588b5b8966587f46b"
format_stamp: "Formatted at 2026-08-06 23:56:23 on dist-test-slave-nj21"
server_key: "014548423073ff81b14b4ac926dbb6b5"
server_key_iv: "8645e17c8165c22d28168e3a324467c3"
server_key_version: "encryptionkey@0"
I20260806 23:56:23.192660 9962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:56:23.195206 9989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:23.196574 9962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 23:56:23.196704 9962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "42fcbedce7794d5588b5b8966587f46b"
format_stamp: "Formatted at 2026-08-06 23:56:23 on dist-test-slave-nj21"
server_key: "014548423073ff81b14b4ac926dbb6b5"
server_key_iv: "8645e17c8165c22d28168e3a324467c3"
server_key_version: "encryptionkey@0"
I20260806 23:56:23.196820 9962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:23.211038 9962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:23.214617 9962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:23.214840 9962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:23.223385 9962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40697
I20260806 23:56:23.223644 10075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40697 every 8 connection(s)
I20260806 23:56:23.224529 9962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:23.227440 10076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:23.228451 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 9962
I20260806 23:56:23.228600 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:23.228928 11142 external_mini_cluster.cc:1428] Setting key 2b6f62681a59d5ab9b6160e30cf19c9f
I20260806 23:56:23.234283 10076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b: Bootstrap starting.
I20260806 23:56:23.237142 10076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:23.238059 10076 log.cc:824] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:23.239930 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:23.230351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:36283 (local address 127.0.0.1:40697)
0806 23:56:23.230778 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:23.230792 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:23.230828 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:56:23.231482 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:23.231499 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:23.231899 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:23.232174 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:23.232849 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:23.234553 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:23.235860 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:23.236365 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:23.236505 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:23.237547 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:23.237553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:23.237567 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:23.237966 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:23.238024 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:23.238049 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:23.238163 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:23.238525 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 23:56:23.238793 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260806 23:56:23.240262 10076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b: No bootstrap required, opened a new log
I20260806 23:56:23.243577 10076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbedce7794d5588b5b8966587f46b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } }
I20260806 23:56:23.243863 10076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:23.243889 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:23.243919 10076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42fcbedce7794d5588b5b8966587f46b, State: Initialized, Role: FOLLOWER
I20260806 23:56:23.244374 10076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbedce7794d5588b5b8966587f46b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } }
I20260806 23:56:23.244522 10076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:23.244573 10076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:23.244657 10076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:23.246191 10076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbedce7794d5588b5b8966587f46b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } }
I20260806 23:56:23.246562 10076 leader_election.cc:302] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42fcbedce7794d5588b5b8966587f46b; no voters:
I20260806 23:56:23.246948 10076 leader_election.cc:288] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:23.247045 10088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:23.247344 10088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [term 1 LEADER]: Becoming Leader. State: Replica: 42fcbedce7794d5588b5b8966587f46b, State: Running, Role: LEADER
I20260806 23:56:23.247661 10088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbedce7794d5588b5b8966587f46b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } }
I20260806 23:56:23.248353 10076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:23.249357 10091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42fcbedce7794d5588b5b8966587f46b. Latest consensus state: current_term: 1 leader_uuid: "42fcbedce7794d5588b5b8966587f46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbedce7794d5588b5b8966587f46b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } } }
I20260806 23:56:23.249497 10091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:23.249927 10090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42fcbedce7794d5588b5b8966587f46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42fcbedce7794d5588b5b8966587f46b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40697 } } }
I20260806 23:56:23.250027 10090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:23.250347 10106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:23.251312 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:23.246434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:40165 (local address 127.0.0.1:40697)
0806 23:56:23.246570 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:23.246575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:23.246732 (+ 157us) server_negotiation.cc:407] Connection header received
0806 23:56:23.246906 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:23.246910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:23.246967 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:23.247051 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:23.247404 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:23.248157 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:23.249308 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:23.249648 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:23.249754 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:23.250667 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:23.250672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:23.250675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:23.250689 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:23.250718 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:23.250731 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:23.250818 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:23.251127 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 23:56:23.251178 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 23:56:23.253983 10106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:23.260289 10106 catalog_manager.cc:1347] Generated new cluster ID: f8890710e12040858dbb47db75b219f6
I20260806 23:56:23.260360 10106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:23.269291 10106 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:23.270264 10106 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:23.274010 10106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42fcbedce7794d5588b5b8966587f46b: Generated new TSK 0
I20260806 23:56:23.274791 10106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:23.286214 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 9962
W20260806 23:56:23.299450 9944 connection.cc:535] client connection to 127.0.0.1:40697 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40697): Connection reset by peer (error 104)
2026-08-06T23:56:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:23 dist-test-slave-nj21 krb5kdc[10123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:23 dist-test-slave-nj21 krb5kdc[10123](info): set up 2 sockets
Aug 06 23:56:23 dist-test-slave-nj21 krb5kdc[10123](info): commencing operation
krb5kdc: starting...
W20260806 23:56:24.437507 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:24 dist-test-slave-nj21 krb5kdc[10123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:24.568320 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:24.688009 10141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:24.688279 10141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:24.688321 10141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:24.692312 10141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:24.692373 10141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:24.692390 10141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:24.692406 10141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:24.692425 10141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:24.697710 10141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:24.698800 10141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:24.699863 10141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:56:24.708073 10141 server_base.cc:1034] running on GCE node
W20260806 23:56:24.708022 10153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:24.708048 10149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:24.708016 10151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:24.708894 10141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:24.710038 10141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:24.711207 10141 hybrid_clock.cc:648] HybridClock initialized: now 1786060584711180 us; error 43 us; skew 500 ppm
Aug 06 23:56:24 dist-test-slave-nj21 krb5kdc[10123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:24.713943 10141 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:24.714785 10141 webserver.cc:460] Webserver started at http://127.0.0.1:36667/ using document root <none> and password file <none>
I20260806 23:56:24.715306 10141 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:24.715364 10141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:24.715534 10141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:24.717625 10141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "6ff472d6c523475fb50fdbf192a1ceb2"
format_stamp: "Formatted at 2026-08-06 23:56:24 on dist-test-slave-nj21"
server_key: "ce306026bb045b9e935d4533666bc6d6"
server_key_iv: "5bd9cb0e0b921011b5b20eaf31189c40"
server_key_version: "encryptionkey@0"
I20260806 23:56:24.718142 10141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "6ff472d6c523475fb50fdbf192a1ceb2"
format_stamp: "Formatted at 2026-08-06 23:56:24 on dist-test-slave-nj21"
server_key: "ce306026bb045b9e935d4533666bc6d6"
server_key_iv: "5bd9cb0e0b921011b5b20eaf31189c40"
server_key_version: "encryptionkey@0"
I20260806 23:56:24.721951 10141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:24.724398 10167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:24.725673 10141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:24.725792 10141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "6ff472d6c523475fb50fdbf192a1ceb2"
format_stamp: "Formatted at 2026-08-06 23:56:24 on dist-test-slave-nj21"
server_key: "ce306026bb045b9e935d4533666bc6d6"
server_key_iv: "5bd9cb0e0b921011b5b20eaf31189c40"
server_key_version: "encryptionkey@0"
I20260806 23:56:24.725883 10141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:24.730397 10141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:24.733346 10141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:24.733494 10141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:24.740540 10141 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43549
I20260806 23:56:24.740607 10251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43549 every 8 connection(s)
I20260806 23:56:24.741534 10141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:24.744330 10253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:24.746699 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 10141
I20260806 23:56:24.746817 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:24.747083 11142 external_mini_cluster.cc:1428] Setting key e41a4a0c912e71b4b9776f194c41ecfc
I20260806 23:56:24.750720 10253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2: Bootstrap starting.
I20260806 23:56:24.752763 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:24.748492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:60965 (local address 127.0.0.1:43549)
0806 23:56:24.748954 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:24.748973 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:24.749016 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:56:24.749849 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:24.749872 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:24.750235 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:24.750594 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:24.751262 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:24.751268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:24.751287 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:24.751299 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:24.751359 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:24.751385 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:24.751517 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:24.751644 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 23:56:24.751830 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260806 23:56:24.753494 10253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:24.754220 10253 log.cc:824] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:24.755466 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:24.756434 10253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2: No bootstrap required, opened a new log
I20260806 23:56:24.759274 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:24.757790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:55883 (local address 127.0.0.1:43549)
0806 23:56:24.757920 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:24.757924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:24.758085 (+ 161us) server_negotiation.cc:407] Connection header received
0806 23:56:24.758211 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:24.758213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:24.758245 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:24.758309 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:24.758939 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:24.758942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:24.758944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:24.758953 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:24.758977 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:24.758987 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:24.759050 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:24.759131 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 23:56:24.759163 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 23:56:24.760842 10253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff472d6c523475fb50fdbf192a1ceb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260806 23:56:24.761072 10253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:24.761130 10253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff472d6c523475fb50fdbf192a1ceb2, State: Initialized, Role: FOLLOWER
I20260806 23:56:24.761543 10253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff472d6c523475fb50fdbf192a1ceb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260806 23:56:24.761672 10253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:24.761737 10253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:24.761839 10253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:24.762775 10253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff472d6c523475fb50fdbf192a1ceb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260806 23:56:24.763101 10253 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ff472d6c523475fb50fdbf192a1ceb2; no voters:
I20260806 23:56:24.763427 10253 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:24.763535 10264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:24.763778 10264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff472d6c523475fb50fdbf192a1ceb2, State: Running, Role: LEADER
I20260806 23:56:24.764144 10264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff472d6c523475fb50fdbf192a1ceb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260806 23:56:24.764628 10253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:24.765671 10265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ff472d6c523475fb50fdbf192a1ceb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff472d6c523475fb50fdbf192a1ceb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } } }
I20260806 23:56:24.765803 10265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:24.766203 10279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:24.766160 10268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff472d6c523475fb50fdbf192a1ceb2. Latest consensus state: current_term: 1 leader_uuid: "6ff472d6c523475fb50fdbf192a1ceb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff472d6c523475fb50fdbf192a1ceb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } } }
I20260806 23:56:24.766330 10268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:24.769192 10279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:24.774955 10279 catalog_manager.cc:1347] Generated new cluster ID: fbcafecc171948cba0a97057e565f99f
I20260806 23:56:24.775079 10279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:24.800561 10279 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:24.800899 10279 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:24.812077 10279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff472d6c523475fb50fdbf192a1ceb2: Generated new TSK 0
I20260806 23:56:24.813210 10279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:24.823483 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 10141
2026-08-06T23:56:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1535 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:24 dist-test-slave-nj21 krb5kdc[10301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:24 dist-test-slave-nj21 krb5kdc[10301](info): set up 2 sockets
Aug 06 23:56:24 dist-test-slave-nj21 krb5kdc[10301](info): commencing operation
krb5kdc: starting...
W20260806 23:56:25.981056 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:26 dist-test-slave-nj21 krb5kdc[10301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:26.104591 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:26.218760 10317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:26.219053 10317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:26.219095 10317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:26.223372 10317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:26.223443 10317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:26.223462 10317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:26.223479 10317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:26.223492 10317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:26.228935 10317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:26.230118 10317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:26.231197 10317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:56:26.239526 10317 server_base.cc:1034] running on GCE node
W20260806 23:56:26.239552 10327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:26.239790 10324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:26.239845 10329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:26.240450 10317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:26.241564 10317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:26.242715 10317 hybrid_clock.cc:648] HybridClock initialized: now 1786060586242702 us; error 35 us; skew 500 ppm
Aug 06 23:56:26 dist-test-slave-nj21 krb5kdc[10301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:26.245428 10317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:26.248123 10317 webserver.cc:460] Webserver started at http://127.0.0.1:42561/ using document root <none> and password file <none>
I20260806 23:56:26.248692 10317 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:26.248762 10317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:26.248975 10317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:26.251729 10317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "600ddcd2f01a422389bdb6d3daacda6d"
format_stamp: "Formatted at 2026-08-06 23:56:26 on dist-test-slave-nj21"
server_key: "0a3abe47dc1ffe81203bbb041007b22b"
server_key_iv: "08ed8ae908269a410689986ac614fd13"
server_key_version: "encryptionkey@0"
I20260806 23:56:26.252357 10317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "600ddcd2f01a422389bdb6d3daacda6d"
format_stamp: "Formatted at 2026-08-06 23:56:26 on dist-test-slave-nj21"
server_key: "0a3abe47dc1ffe81203bbb041007b22b"
server_key_iv: "08ed8ae908269a410689986ac614fd13"
server_key_version: "encryptionkey@0"
I20260806 23:56:26.256155 10317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:56:26.258562 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:26.259768 10317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:26.259893 10317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "600ddcd2f01a422389bdb6d3daacda6d"
format_stamp: "Formatted at 2026-08-06 23:56:26 on dist-test-slave-nj21"
server_key: "0a3abe47dc1ffe81203bbb041007b22b"
server_key_iv: "08ed8ae908269a410689986ac614fd13"
server_key_version: "encryptionkey@0"
I20260806 23:56:26.259986 10317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:26.280689 10317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:26.283842 10317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:26.284042 10317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:26.292098 10317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44311
I20260806 23:56:26.292125 10418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44311 every 8 connection(s)
I20260806 23:56:26.293076 10317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:26.296229 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:26.302022 10420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d: Bootstrap starting.
I20260806 23:56:26.302083 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 10317
I20260806 23:56:26.302203 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:26.302470 11142 external_mini_cluster.cc:1428] Setting key 2010946df635d4ab0a11912e3a2d9801
I20260806 23:56:26.304701 10420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:26.305579 10420 log.cc:824] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:26.307852 10420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d: No bootstrap required, opened a new log
Aug 06 23:56:26 dist-test-slave-nj21 krb5kdc[10301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 23:56:26.312196 10420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600ddcd2f01a422389bdb6d3daacda6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } }
I20260806 23:56:26.312433 10420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:26.312492 10420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600ddcd2f01a422389bdb6d3daacda6d, State: Initialized, Role: FOLLOWER
I20260806 23:56:26.313009 10420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600ddcd2f01a422389bdb6d3daacda6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } }
I20260806 23:56:26.313176 10420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:26.313246 10420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:26.313357 10420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:26.314255 10420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600ddcd2f01a422389bdb6d3daacda6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } }
I20260806 23:56:26.314581 10420 leader_election.cc:302] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 600ddcd2f01a422389bdb6d3daacda6d; no voters:
I20260806 23:56:26.314886 10420 leader_election.cc:288] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:26.314999 10430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:26.315174 10430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [term 1 LEADER]: Becoming Leader. State: Replica: 600ddcd2f01a422389bdb6d3daacda6d, State: Running, Role: LEADER
I20260806 23:56:26.315534 10430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600ddcd2f01a422389bdb6d3daacda6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } }
I20260806 23:56:26.316267 10420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:26.317559 10431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "600ddcd2f01a422389bdb6d3daacda6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600ddcd2f01a422389bdb6d3daacda6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } } }
I20260806 23:56:26.317754 10431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:26.318071 10432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 600ddcd2f01a422389bdb6d3daacda6d. Latest consensus state: current_term: 1 leader_uuid: "600ddcd2f01a422389bdb6d3daacda6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600ddcd2f01a422389bdb6d3daacda6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44311 } } }
I20260806 23:56:26.318130 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:26.303813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:39725 (local address 127.0.0.1:44311)
0806 23:56:26.304238 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:26.304257 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:26.304308 (+ 51us) server_negotiation.cc:407] Connection header received
0806 23:56:26.305143 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:26.305165 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:26.305600 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:26.305966 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:26.306745 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:26.308199 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:26.309039 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:26.310092 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:26.310263 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:26.312701 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:26.312709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:26.312725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:26.312756 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:26.314411 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:26.314906 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:26.314912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:26.314918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:26.315071 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:26.315430 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:26.315434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:26.315436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:26.315936 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:26.316089 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:26.316551 (+ 462us) server_negotiation.cc:299] Negotiation successful
0806 23:56:26.316774 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260806 23:56:26.318151 10432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:26.318738 10441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:26.321394 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:26.322748 10441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:26.329070 10441 catalog_manager.cc:1347] Generated new cluster ID: 6119acc89083425cb691d356652cc479
I20260806 23:56:26.329208 10441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:26.332561 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:26.323532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:45157 (local address 127.0.0.1:44311)
0806 23:56:26.323639 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:26.323643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:26.323993 (+ 350us) server_negotiation.cc:407] Connection header received
0806 23:56:26.324092 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:26.324095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:26.324142 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:26.324207 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:26.324857 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:26.325678 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:26.326889 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:26.327196 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:26.327294 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:26.328994 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:26.328998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:26.329001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:26.329028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:26.330763 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:26.331267 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:26.331271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:26.331273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:26.331364 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:26.331655 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:26.331658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:26.331660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:26.331907 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:26.332054 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:26.332283 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 23:56:26.332396 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 23:56:26.336378 10441 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:26.337508 10441 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:26.341845 10441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 600ddcd2f01a422389bdb6d3daacda6d: Generated new TSK 0
I20260806 23:56:26.342550 10441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:26.347182 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 10317
W20260806 23:56:26.370141 10296 connection.cc:535] client connection to 127.0.0.1:44311 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44311): Connection reset by peer (error 104)
2026-08-06T23:56:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1534 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:26 dist-test-slave-nj21 krb5kdc[10465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:26 dist-test-slave-nj21 krb5kdc[10465](info): set up 2 sockets
Aug 06 23:56:26 dist-test-slave-nj21 krb5kdc[10465](info): commencing operation
krb5kdc: starting...
W20260806 23:56:27.502543 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:27 dist-test-slave-nj21 krb5kdc[10465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:27.632218 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:27.751631 10481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:27.751950 10481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:27.752012 10481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:27.756222 10481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:27.756441 10481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:27.756512 10481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:27.756539 10481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:27.756570 10481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:27.762267 10481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:27.763392 10481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:27.764530 10481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:27.773187 10491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:27.773367 10481 server_base.cc:1034] running on GCE node
W20260806 23:56:27.773187 10488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:27.773522 10494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:27.774060 10481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:27.775175 10481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:27.776467 10481 hybrid_clock.cc:648] HybridClock initialized: now 1786060587776431 us; error 25 us; skew 500 ppm
Aug 06 23:56:27 dist-test-slave-nj21 krb5kdc[10465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:27.783241 10481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:27.784317 10481 webserver.cc:460] Webserver started at http://127.0.0.1:39717/ using document root <none> and password file <none>
I20260806 23:56:27.784900 10481 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:27.784967 10481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:27.785167 10481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:27.787294 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "9de5b1335a8a471faa14e8ce6a6796d5"
format_stamp: "Formatted at 2026-08-06 23:56:27 on dist-test-slave-nj21"
server_key: "00475353999529f53ef78fc39513168d"
server_key_iv: "1807b21153b43c91a5ac15afe97d4d6b"
server_key_version: "encryptionkey@0"
I20260806 23:56:27.787900 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "9de5b1335a8a471faa14e8ce6a6796d5"
format_stamp: "Formatted at 2026-08-06 23:56:27 on dist-test-slave-nj21"
server_key: "00475353999529f53ef78fc39513168d"
server_key_iv: "1807b21153b43c91a5ac15afe97d4d6b"
server_key_version: "encryptionkey@0"
I20260806 23:56:27.791778 10481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:56:27.794250 10504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:27.795532 10481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:27.795656 10481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "9de5b1335a8a471faa14e8ce6a6796d5"
format_stamp: "Formatted at 2026-08-06 23:56:27 on dist-test-slave-nj21"
server_key: "00475353999529f53ef78fc39513168d"
server_key_iv: "1807b21153b43c91a5ac15afe97d4d6b"
server_key_version: "encryptionkey@0"
I20260806 23:56:27.795806 10481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:27.824036 10481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:27.827008 10481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:27.827186 10481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:27.835506 10481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40971
I20260806 23:56:27.835527 10583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40971 every 8 connection(s)
I20260806 23:56:27.836527 10481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:27.839354 10584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:27.840394 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 10481
I20260806 23:56:27.840533 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:27.840811 11142 external_mini_cluster.cc:1428] Setting key 2a6d7979b3bf03df14dda5e9bf393ca7
I20260806 23:56:27.845499 10584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5: Bootstrap starting.
I20260806 23:56:27.848115 10584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:27.848997 10584 log.cc:824] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:27.850243 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:27.842128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:48607 (local address 127.0.0.1:40971)
0806 23:56:27.842534 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:27.842550 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:27.842586 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:56:27.843212 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:27.843227 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:27.843571 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:27.843885 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:27.844220 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:27.845175 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:27.846329 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:27.846798 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:27.846911 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:27.847869 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:27.847874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:27.847886 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:27.848280 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:27.848532 (+ 252us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:27.848582 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:27.848792 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:27.849097 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 23:56:27.849269 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260806 23:56:27.851094 10584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5: No bootstrap required, opened a new log
I20260806 23:56:27.853173 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:27.855506 10584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de5b1335a8a471faa14e8ce6a6796d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } }
I20260806 23:56:27.855823 10584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:27.855888 10584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de5b1335a8a471faa14e8ce6a6796d5, State: Initialized, Role: FOLLOWER
I20260806 23:56:27.856308 10584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de5b1335a8a471faa14e8ce6a6796d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } }
I20260806 23:56:27.856431 10584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:27.856488 10584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:27.856583 10584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:27.857491 10584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de5b1335a8a471faa14e8ce6a6796d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } }
I20260806 23:56:27.857823 10584 leader_election.cc:302] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9de5b1335a8a471faa14e8ce6a6796d5; no voters:
I20260806 23:56:27.858107 10584 leader_election.cc:288] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:27.858230 10597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:27.858467 10597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9de5b1335a8a471faa14e8ce6a6796d5, State: Running, Role: LEADER
I20260806 23:56:27.858850 10597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de5b1335a8a471faa14e8ce6a6796d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } }
I20260806 23:56:27.859555 10584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:27.860154 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:27.855456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:52765 (local address 127.0.0.1:40971)
0806 23:56:27.855556 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:27.855560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:27.855731 (+ 171us) server_negotiation.cc:407] Connection header received
0806 23:56:27.855867 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:27.855870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:27.855926 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:27.856000 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:27.856252 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:27.856877 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:27.858004 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:27.858436 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:27.858543 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:27.859617 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:27.859622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:27.859625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:27.859641 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:27.859696 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:27.859714 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:27.859817 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:27.859967 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 23:56:27.860007 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 23:56:27.860579 10601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9de5b1335a8a471faa14e8ce6a6796d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de5b1335a8a471faa14e8ce6a6796d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } } }
I20260806 23:56:27.860720 10601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:27.861132 10602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de5b1335a8a471faa14e8ce6a6796d5. Latest consensus state: current_term: 1 leader_uuid: "9de5b1335a8a471faa14e8ce6a6796d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de5b1335a8a471faa14e8ce6a6796d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } } }
I20260806 23:56:27.861295 10602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5 [sys.catalog]: This master's current role is: LEADER
W20260806 23:56:27.864568 10621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:27.864637 10621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:27.864703 10611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:27.865466 10611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:27.870981 10611 catalog_manager.cc:1347] Generated new cluster ID: b714de56f18c456987aa3f043d91c294
I20260806 23:56:27.871055 10611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:27.894610 10611 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:27.895993 10611 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:27.915300 10611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9de5b1335a8a471faa14e8ce6a6796d5: Generated new TSK 0
I20260806 23:56:27.916237 10611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:27.930362 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 10481
W20260806 23:56:27.949205 10462 connection.cc:535] client connection to 127.0.0.1:40971 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40971): Connection reset by peer (error 104)
2026-08-06T23:56:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1578 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:27 dist-test-slave-nj21 krb5kdc[10633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:27 dist-test-slave-nj21 krb5kdc[10633](info): set up 2 sockets
Aug 06 23:56:27 dist-test-slave-nj21 krb5kdc[10633](info): commencing operation
krb5kdc: starting...
W20260806 23:56:29.071834 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:29 dist-test-slave-nj21 krb5kdc[10633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 23:56:29.201421 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:29.318282 10659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:29.318599 10659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:29.318662 10659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:29.322763 10659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:29.322839 10659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:29.322867 10659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:29.322889 10659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:29.322911 10659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:29.328388 10659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:29.329545 10659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:29.330571 10659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:29.339135 10668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:29.339131 10665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:29.339323 10664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:29.339350 10659 server_base.cc:1034] running on GCE node
I20260806 23:56:29.340126 10659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:29.341264 10659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:29.342446 10659 hybrid_clock.cc:648] HybridClock initialized: now 1786060589342422 us; error 34 us; skew 500 ppm
Aug 06 23:56:29 dist-test-slave-nj21 krb5kdc[10633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:29.345319 10659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 23:56:29.346242 10659 webserver.cc:460] Webserver started at http://127.0.0.1:42193/ using document root <none> and password file <none>
I20260806 23:56:29.346796 10659 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:29.346863 10659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:29.347051 10659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:29.349174 10659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "9842645ddce743af87cfdb941d649bb4"
format_stamp: "Formatted at 2026-08-06 23:56:29 on dist-test-slave-nj21"
server_key: "91a20d47dcc9266d265893cafdc7578a"
server_key_iv: "5e184c17e1f4f8d65ea25cf00148b13c"
server_key_version: "encryptionkey@0"
I20260806 23:56:29.349745 10659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "9842645ddce743af87cfdb941d649bb4"
format_stamp: "Formatted at 2026-08-06 23:56:29 on dist-test-slave-nj21"
server_key: "91a20d47dcc9266d265893cafdc7578a"
server_key_iv: "5e184c17e1f4f8d65ea25cf00148b13c"
server_key_version: "encryptionkey@0"
I20260806 23:56:29.353447 10659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 23:56:29.355871 10681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:29.357254 10659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:56:29.357381 10659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "9842645ddce743af87cfdb941d649bb4"
format_stamp: "Formatted at 2026-08-06 23:56:29 on dist-test-slave-nj21"
server_key: "91a20d47dcc9266d265893cafdc7578a"
server_key_iv: "5e184c17e1f4f8d65ea25cf00148b13c"
server_key_version: "encryptionkey@0"
I20260806 23:56:29.357488 10659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:29.362128 10659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:29.365386 10659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:29.365579 10659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:29.372830 10659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43705
I20260806 23:56:29.372872 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43705 every 8 connection(s)
I20260806 23:56:29.373791 10659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:29.376765 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:29.380472 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 10659
I20260806 23:56:29.380604 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:29.380858 11142 external_mini_cluster.cc:1428] Setting key bb88276df6e30c470c72b9e0d7ed7da0
I20260806 23:56:29.382946 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4: Bootstrap starting.
I20260806 23:56:29.385643 10759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:29.386612 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:29.382494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:35959 (local address 127.0.0.1:43705)
0806 23:56:29.383016 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:29.383033 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:29.383076 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:56:29.383813 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:29.383832 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:29.384118 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:29.384422 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:29.385272 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:29.385277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:29.385289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:29.385301 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:29.385358 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:29.385381 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:29.385481 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:29.385744 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 23:56:29.385908 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":168,"threads_started":1}
I20260806 23:56:29.386626 10759 log.cc:824] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:29.389410 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4: No bootstrap required, opened a new log
I20260806 23:56:29.390004 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:29.393779 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:29.392052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:50569 (local address 127.0.0.1:43705)
0806 23:56:29.392182 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:29.392187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:29.392416 (+ 229us) server_negotiation.cc:407] Connection header received
0806 23:56:29.392524 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:29.392527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:29.392566 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:29.392632 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:29.393390 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:29.393394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:29.393398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:29.393411 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:29.393443 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:29.393456 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:29.393530 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:29.393611 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 23:56:29.393655 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 23:56:29.393635 10759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9842645ddce743af87cfdb941d649bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260806 23:56:29.393972 10759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:29.394042 10759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9842645ddce743af87cfdb941d649bb4, State: Initialized, Role: FOLLOWER
I20260806 23:56:29.394490 10759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9842645ddce743af87cfdb941d649bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260806 23:56:29.394634 10759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:29.394707 10759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:29.394817 10759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:29.395880 10759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9842645ddce743af87cfdb941d649bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260806 23:56:29.396229 10759 leader_election.cc:302] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9842645ddce743af87cfdb941d649bb4; no voters:
I20260806 23:56:29.396524 10759 leader_election.cc:288] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:29.396613 10770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:29.396848 10770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 9842645ddce743af87cfdb941d649bb4, State: Running, Role: LEADER
I20260806 23:56:29.397256 10770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9842645ddce743af87cfdb941d649bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260806 23:56:29.397763 10759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:29.399235 10771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9842645ddce743af87cfdb941d649bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9842645ddce743af87cfdb941d649bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } } }
I20260806 23:56:29.399283 10772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9842645ddce743af87cfdb941d649bb4. Latest consensus state: current_term: 1 leader_uuid: "9842645ddce743af87cfdb941d649bb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9842645ddce743af87cfdb941d649bb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } } }
I20260806 23:56:29.399379 10771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:29.399379 10772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:29.399817 10785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:29.403426 10785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:29.409124 10785 catalog_manager.cc:1347] Generated new cluster ID: 097dec99d4ca48dbac49e2a7cf0896a9
I20260806 23:56:29.409260 10785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:29.420459 10785 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:29.420835 10785 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:29.434995 10785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9842645ddce743af87cfdb941d649bb4: Generated new TSK 0
I20260806 23:56:29.436000 10785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:29.444895 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 10659
2026-08-06T23:56:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1520 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:29 dist-test-slave-nj21 krb5kdc[10804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:29 dist-test-slave-nj21 krb5kdc[10804](info): set up 2 sockets
Aug 06 23:56:29 dist-test-slave-nj21 krb5kdc[10804](info): commencing operation
krb5kdc: starting...
W20260806 23:56:30.600113 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:30 dist-test-slave-nj21 krb5kdc[10804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:30.726565 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:32845
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:32845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:30.848346 10820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:30.848650 10820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:30.848713 10820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:30.853058 10820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:30.853150 10820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:30.853180 10820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:30.853202 10820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:30.853231 10820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:30.859978 10820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:32845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:32845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:30.861172 10820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:30.862253 10820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:30.870754 10829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:30.870908 10830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:30.870980 10832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:30.871281 10820 server_base.cc:1034] running on GCE node
I20260806 23:56:30.871901 10820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:30.873039 10820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:30.874231 10820 hybrid_clock.cc:648] HybridClock initialized: now 1786060590874163 us; error 80 us; skew 500 ppm
Aug 06 23:56:30 dist-test-slave-nj21 krb5kdc[10804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060590, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:30.876946 10820 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:30.877851 10820 webserver.cc:460] Webserver started at http://127.10.225.190:44623/ using document root <none> and password file <none>
I20260806 23:56:30.878403 10820 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:30.878469 10820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:30.878679 10820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:30.880842 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "d5482f37f4264f99a5a1f8360e30ec07"
format_stamp: "Formatted at 2026-08-06 23:56:30 on dist-test-slave-nj21"
server_key: "e0d7502924be621d770ef80540eba023"
server_key_iv: "ff97f3e5719f47a04a1e3f53cdc52c53"
server_key_version: "encryptionkey@0"
I20260806 23:56:30.881392 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "d5482f37f4264f99a5a1f8360e30ec07"
format_stamp: "Formatted at 2026-08-06 23:56:30 on dist-test-slave-nj21"
server_key: "e0d7502924be621d770ef80540eba023"
server_key_iv: "ff97f3e5719f47a04a1e3f53cdc52c53"
server_key_version: "encryptionkey@0"
I20260806 23:56:30.885191 10820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:56:30.887588 10841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:30.889209 10820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:30.889338 10820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "d5482f37f4264f99a5a1f8360e30ec07"
format_stamp: "Formatted at 2026-08-06 23:56:30 on dist-test-slave-nj21"
server_key: "e0d7502924be621d770ef80540eba023"
server_key_iv: "ff97f3e5719f47a04a1e3f53cdc52c53"
server_key_version: "encryptionkey@0"
I20260806 23:56:30.889441 10820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:30.915863 10820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:30.918932 10820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:30.919124 10820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:30.927433 10820 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:32845
I20260806 23:56:30.927479 10909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:32845 every 8 connection(s)
I20260806 23:56:30.928434 10820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:30.931260 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:30.935231 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 10820
I20260806 23:56:30.935405 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:30.935714 11142 external_mini_cluster.cc:1428] Setting key cafd7a030e9448375d24d22f6ac18a09
I20260806 23:56:30.937599 10910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07: Bootstrap starting.
I20260806 23:56:30.939868 10910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:30.940611 10910 log.cc:824] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:30.942495 10910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07: No bootstrap required, opened a new log
Aug 06 23:56:30 dist-test-slave-nj21 krb5kdc[10804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:30.946600 10910 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5482f37f4264f99a5a1f8360e30ec07" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32845 } }
I20260806 23:56:30.946914 10910 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:30.946976 10910 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5482f37f4264f99a5a1f8360e30ec07, State: Initialized, Role: FOLLOWER
I20260806 23:56:30.947515 10910 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5482f37f4264f99a5a1f8360e30ec07" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32845 } }
I20260806 23:56:30.947710 10910 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:30.947789 10910 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:30.947891 10910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:30.949083 10910 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5482f37f4264f99a5a1f8360e30ec07" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32845 } }
I20260806 23:56:30.949496 10910 leader_election.cc:302] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5482f37f4264f99a5a1f8360e30ec07; no voters:
I20260806 23:56:30.949868 10910 leader_election.cc:288] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:30.950119 10917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:30.950410 10917 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [term 1 LEADER]: Becoming Leader. State: Replica: d5482f37f4264f99a5a1f8360e30ec07, State: Running, Role: LEADER
I20260806 23:56:30.950759 10917 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5482f37f4264f99a5a1f8360e30ec07" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32845 } }
I20260806 23:56:30.951000 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:30.937241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60463 (local address 127.10.225.190:32845)
0806 23:56:30.937743 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:30.937765 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:30.937815 (+ 50us) server_negotiation.cc:407] Connection header received
0806 23:56:30.938469 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:30.938486 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:30.938831 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:30.939137 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:30.940051 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:30.941335 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:30.942186 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:30.942757 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:30.942909 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:30.945678 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:30.945687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:30.945705 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:30.945738 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:30.947630 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:30.948230 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:30.948236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:30.948242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:30.948380 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:30.948665 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:30.948669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:30.948670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:30.949233 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:30.949453 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:30.949757 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 23:56:30.950013 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":148,"threads_started":1}
I20260806 23:56:30.951213 10910 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:30.952665 10919 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5482f37f4264f99a5a1f8360e30ec07. Latest consensus state: current_term: 1 leader_uuid: "d5482f37f4264f99a5a1f8360e30ec07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5482f37f4264f99a5a1f8360e30ec07" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32845 } } }
I20260806 23:56:30.952816 10919 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:30.953269 10918 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5482f37f4264f99a5a1f8360e30ec07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5482f37f4264f99a5a1f8360e30ec07" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 32845 } } }
I20260806 23:56:30.953366 10918 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:30.953929 10930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:30.954510 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:30.957299 10930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:30.963610 10930 catalog_manager.cc:1347] Generated new cluster ID: 62d36f2e95364e6d92fcaf2f9bd9bd97
I20260806 23:56:30.963709 10930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:30.965116 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:30.956593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34967 (local address 127.10.225.190:32845)
0806 23:56:30.956709 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:30.956714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:30.957055 (+ 341us) server_negotiation.cc:407] Connection header received
0806 23:56:30.957160 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:30.957163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:30.957207 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:30.957271 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:30.957939 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:30.958454 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:30.959226 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:30.959515 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:30.959610 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:30.961370 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:30.961374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:30.961377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:30.961402 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:30.963249 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:30.963873 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:30.963877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:30.963879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:30.963960 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:30.964280 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:30.964284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:30.964286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:30.964471 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:30.964619 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:30.964866 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 23:56:30.964961 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":43}
I20260806 23:56:31.007961 10930 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:31.008910 10930 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:31.017155 10930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5482f37f4264f99a5a1f8360e30ec07: Generated new TSK 0
I20260806 23:56:31.017853 10930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:31.031831 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 10820
W20260806 23:56:31.050232 10800 connection.cc:535] client connection to 127.10.225.190:32845 recv error: Network error: failed to read from TLS socket (remote: 127.10.225.190:32845): Connection reset by peer (error 104)
2026-08-06T23:56:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1580 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:31 dist-test-slave-nj21 krb5kdc[10952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:31 dist-test-slave-nj21 krb5kdc[10952](info): set up 2 sockets
Aug 06 23:56:31 dist-test-slave-nj21 krb5kdc[10952](info): commencing operation
krb5kdc: starting...
W20260806 23:56:32.182407 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:32 dist-test-slave-nj21 krb5kdc[10952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:32.312381 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:39829
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:42085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:39829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:32.428445 10968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:32.428745 10968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:32.428805 10968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:32.433071 10968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:32.433156 10968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:32.433182 10968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:32.433203 10968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:32.433229 10968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:32.438745 10968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:39829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:39829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:32.439939 10968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:32.440990 10968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:32.449334 10984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:32.449344 10979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:32.449337 10981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:32.449637 10968 server_base.cc:1034] running on GCE node
I20260806 23:56:32.450270 10968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:32.451637 10968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:32.452865 10968 hybrid_clock.cc:648] HybridClock initialized: now 1786060592452842 us; error 39 us; skew 500 ppm
Aug 06 23:56:32 dist-test-slave-nj21 krb5kdc[10952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060592, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:32.455713 10968 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:32.456679 10968 webserver.cc:460] Webserver started at http://127.10.225.190:37519/ using document root <none> and password file <none>
I20260806 23:56:32.457212 10968 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:32.457279 10968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:32.457468 10968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:32.459510 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "9bd1a76a051c4fd4aba52eb4f01075c5"
format_stamp: "Formatted at 2026-08-06 23:56:32 on dist-test-slave-nj21"
server_key: "2ae268485edb5c9af2a0e5e14b15d150"
server_key_iv: "97e4e282c5ba417a3c77b060744800c5"
server_key_version: "encryptionkey@0"
I20260806 23:56:32.460090 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "9bd1a76a051c4fd4aba52eb4f01075c5"
format_stamp: "Formatted at 2026-08-06 23:56:32 on dist-test-slave-nj21"
server_key: "2ae268485edb5c9af2a0e5e14b15d150"
server_key_iv: "97e4e282c5ba417a3c77b060744800c5"
server_key_version: "encryptionkey@0"
I20260806 23:56:32.464483 10968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 23:56:32.467202 10999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:32.468894 10968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:32.469028 10968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "9bd1a76a051c4fd4aba52eb4f01075c5"
format_stamp: "Formatted at 2026-08-06 23:56:32 on dist-test-slave-nj21"
server_key: "2ae268485edb5c9af2a0e5e14b15d150"
server_key_iv: "97e4e282c5ba417a3c77b060744800c5"
server_key_version: "encryptionkey@0"
I20260806 23:56:32.469185 10968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:32.495458 10968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:32.498966 10968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:32.499185 10968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:32.507531 10968 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:39829
I20260806 23:56:32.507577 11074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:39829 every 8 connection(s)
I20260806 23:56:32.508564 10968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:32.510073 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 10968
I20260806 23:56:32.510207 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:32.510476 11142 external_mini_cluster.cc:1428] Setting key 00c8426274f176b0d88acfcb613ffb7a
I20260806 23:56:32.511840 11076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:32.519541 11076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5: Bootstrap starting.
I20260806 23:56:32.523013 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:32.512259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50247 (local address 127.10.225.190:39829)
0806 23:56:32.512959 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:32.512981 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:32.513028 (+ 47us) server_negotiation.cc:407] Connection header received
0806 23:56:32.513921 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:32.513946 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:32.514410 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:32.514802 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:32.515270 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:32.516807 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:32.517906 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:32.518644 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:32.518837 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:32.519918 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:32.519927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:32.519950 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:32.520644 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:32.520723 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:32.520760 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:32.520922 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:32.521131 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 23:56:32.521377 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":198,"threads_started":1}
I20260806 23:56:32.523583 11076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:32.524597 11076 log.cc:824] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:32.527565 11076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5: No bootstrap required, opened a new log
I20260806 23:56:32.527741 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:32.532227 11076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1a76a051c4fd4aba52eb4f01075c5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39829 } }
I20260806 23:56:32.532541 11076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:32.532596 11076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd1a76a051c4fd4aba52eb4f01075c5, State: Initialized, Role: FOLLOWER
I20260806 23:56:32.533291 11076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1a76a051c4fd4aba52eb4f01075c5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39829 } }
I20260806 23:56:32.533540 11076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:32.533627 11076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:32.533759 11076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:32.534232 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:32.529945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43981 (local address 127.10.225.190:39829)
0806 23:56:32.530064 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:32.530069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:32.530146 (+ 77us) server_negotiation.cc:407] Connection header received
0806 23:56:32.530278 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:32.530281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:32.530339 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:32.530436 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:32.530676 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:32.531480 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:32.532351 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:32.532680 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:32.532783 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:32.533630 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:32.533635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:32.533639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:32.533659 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:32.533708 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:32.533734 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:32.533875 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:32.534024 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 23:56:32.534079 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 23:56:32.535310 11076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1a76a051c4fd4aba52eb4f01075c5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39829 } }
I20260806 23:56:32.535775 11076 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd1a76a051c4fd4aba52eb4f01075c5; no voters:
I20260806 23:56:32.536164 11076 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:32.536382 11087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:32.536756 11087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd1a76a051c4fd4aba52eb4f01075c5, State: Running, Role: LEADER
I20260806 23:56:32.537158 11087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1a76a051c4fd4aba52eb4f01075c5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39829 } }
I20260806 23:56:32.537756 11076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:32.538928 11089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd1a76a051c4fd4aba52eb4f01075c5. Latest consensus state: current_term: 1 leader_uuid: "9bd1a76a051c4fd4aba52eb4f01075c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1a76a051c4fd4aba52eb4f01075c5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39829 } } }
I20260806 23:56:32.539143 11089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:32.539774 11098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:32.539810 11088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd1a76a051c4fd4aba52eb4f01075c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd1a76a051c4fd4aba52eb4f01075c5" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 39829 } } }
I20260806 23:56:32.539917 11088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:32.543303 11098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:32.550122 11098 catalog_manager.cc:1347] Generated new cluster ID: 767e530997984f418864bef546b178af
I20260806 23:56:32.550294 11098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:32.561017 11098 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:32.562309 11098 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:32.573580 11098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd1a76a051c4fd4aba52eb4f01075c5: Generated new TSK 0
I20260806 23:56:32.574467 11098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:32.580551 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 10968
W20260806 23:56:32.597483 10950 connection.cc:535] client connection to 127.10.225.190:39829 recv error: Network error: failed to read from TLS socket (remote: 127.10.225.190:39829): Connection reset by peer (error 104)
2026-08-06T23:56:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1547 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:32 dist-test-slave-nj21 krb5kdc[11120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:32 dist-test-slave-nj21 krb5kdc[11120](info): set up 2 sockets
Aug 06 23:56:32 dist-test-slave-nj21 krb5kdc[11120](info): commencing operation
krb5kdc: starting...
W20260806 23:56:33.741640 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:33 dist-test-slave-nj21 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:33.868995 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:42317
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:42317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:33.997390 11140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:33.997666 11140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:33.997733 11140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:34.001827 11140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:34.001899 11140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:34.001919 11140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:34.001936 11140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:34.001951 11140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:34.007536 11140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:42317
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:42317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:34.008672 11140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:34.009727 11140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 23:56:34.018183 11140 server_base.cc:1034] running on GCE node
W20260806 23:56:34.018157 11157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:34.018119 11156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:34.018119 11160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:34.018916 11140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:34.020072 11140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:34.021239 11140 hybrid_clock.cc:648] HybridClock initialized: now 1786060594021217 us; error 35 us; skew 500 ppm
Aug 06 23:56:34 dist-test-slave-nj21 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060594, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:34.023976 11140 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:34.024849 11140 webserver.cc:460] Webserver started at http://127.10.225.190:32889/ using document root <none> and password file <none>
I20260806 23:56:34.025391 11140 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:34.025455 11140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:34.025655 11140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:34.027766 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "77c40146e15745ce8a9110e688c1f594"
format_stamp: "Formatted at 2026-08-06 23:56:34 on dist-test-slave-nj21"
server_key: "c2731f55a0bdea07eb0bd3458d027761"
server_key_iv: "c11bc751a4b81991245962a6025c7530"
server_key_version: "encryptionkey@0"
I20260806 23:56:34.028322 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "77c40146e15745ce8a9110e688c1f594"
format_stamp: "Formatted at 2026-08-06 23:56:34 on dist-test-slave-nj21"
server_key: "c2731f55a0bdea07eb0bd3458d027761"
server_key_iv: "c11bc751a4b81991245962a6025c7530"
server_key_version: "encryptionkey@0"
I20260806 23:56:34.032136 11140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 23:56:34.034524 11169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:34.035841 11140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:34.035972 11140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "77c40146e15745ce8a9110e688c1f594"
format_stamp: "Formatted at 2026-08-06 23:56:34 on dist-test-slave-nj21"
server_key: "c2731f55a0bdea07eb0bd3458d027761"
server_key_iv: "c11bc751a4b81991245962a6025c7530"
server_key_version: "encryptionkey@0"
I20260806 23:56:34.036072 11140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:34.040611 11140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:34.043488 11140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:34.043663 11140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:34.050978 11140 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:42317
I20260806 23:56:34.050982 11244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:42317 every 8 connection(s)
I20260806 23:56:34.051976 11140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:34.054769 11245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:34.058029 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11140
I20260806 23:56:34.058185 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:34.058501 11142 external_mini_cluster.cc:1428] Setting key e859357f8a97c02dc121f96fa7285d4b
I20260806 23:56:34.060851 11245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594: Bootstrap starting.
I20260806 23:56:34.063280 11245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:34.063758 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:34.059916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54327 (local address 127.10.225.190:42317)
0806 23:56:34.060337 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:34.060355 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:34.060393 (+ 38us) server_negotiation.cc:407] Connection header received
0806 23:56:34.061052 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:34.061071 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:34.061364 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:34.061632 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:34.062416 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:34.062421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:34.062432 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:34.062442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:34.062498 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:34.062516 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:34.062622 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:34.062707 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 23:56:34.062848 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260806 23:56:34.064088 11245 log.cc:824] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:34.066200 11245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594: No bootstrap required, opened a new log
I20260806 23:56:34.066632 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:34.070636 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:34.068907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53585 (local address 127.10.225.190:42317)
0806 23:56:34.069053 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:34.069057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:34.069268 (+ 211us) server_negotiation.cc:407] Connection header received
0806 23:56:34.069383 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:34.069386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:34.069422 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:34.069505 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:34.070201 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:34.070205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:34.070208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:34.070220 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:34.070246 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:34.070256 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:34.070326 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:34.070408 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 23:56:34.070455 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:56:34.070497 11245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c40146e15745ce8a9110e688c1f594" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42317 } }
I20260806 23:56:34.070808 11245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:34.070873 11245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c40146e15745ce8a9110e688c1f594, State: Initialized, Role: FOLLOWER
I20260806 23:56:34.071307 11245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c40146e15745ce8a9110e688c1f594" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42317 } }
I20260806 23:56:34.071466 11245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:34.071537 11245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:34.071635 11245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:34.072583 11245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c40146e15745ce8a9110e688c1f594" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42317 } }
I20260806 23:56:34.072916 11245 leader_election.cc:302] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c40146e15745ce8a9110e688c1f594; no voters:
I20260806 23:56:34.073189 11245 leader_election.cc:288] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:34.073392 11262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:34.073589 11262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [term 1 LEADER]: Becoming Leader. State: Replica: 77c40146e15745ce8a9110e688c1f594, State: Running, Role: LEADER
I20260806 23:56:34.073945 11262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c40146e15745ce8a9110e688c1f594" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42317 } }
I20260806 23:56:34.074398 11245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:34.076198 11263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c40146e15745ce8a9110e688c1f594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c40146e15745ce8a9110e688c1f594" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42317 } } }
I20260806 23:56:34.076335 11263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:34.076192 11265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c40146e15745ce8a9110e688c1f594. Latest consensus state: current_term: 1 leader_uuid: "77c40146e15745ce8a9110e688c1f594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c40146e15745ce8a9110e688c1f594" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42317 } } }
I20260806 23:56:34.076771 11265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594 [sys.catalog]: This master's current role is: LEADER
W20260806 23:56:34.079025 11288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:34.079092 11288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:34.079165 11290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:34.079923 11290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:34.085431 11290 catalog_manager.cc:1347] Generated new cluster ID: 429d3bb2ac194c8b8d139136e20298be
I20260806 23:56:34.085498 11290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:34.092710 11290 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:34.093035 11290 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:34.098305 11290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c40146e15745ce8a9110e688c1f594: Generated new TSK 0
I20260806 23:56:34.099030 11290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:34.108786 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11140
2026-08-06T23:56:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1524 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:34 dist-test-slave-nj21 krb5kdc[11299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:34 dist-test-slave-nj21 krb5kdc[11299](info): set up 2 sockets
Aug 06 23:56:34 dist-test-slave-nj21 krb5kdc[11299](info): commencing operation
krb5kdc: starting...
W20260806 23:56:35.254456 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:35 dist-test-slave-nj21 krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:35.382709 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38383
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:38383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:35.500178 11321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:35.500494 11321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:35.500710 11321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:35.505668 11321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:35.505769 11321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:35.505800 11321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:35.505823 11321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:35.505846 11321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:35.511546 11321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:38383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:35.512756 11321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:35.513828 11321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:35.522529 11333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:35.522549 11329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:35.522544 11331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:35.522852 11321 server_base.cc:1034] running on GCE node
I20260806 23:56:35.523502 11321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:35.524664 11321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:35.525844 11321 hybrid_clock.cc:648] HybridClock initialized: now 1786060595525818 us; error 37 us; skew 500 ppm
Aug 06 23:56:35 dist-test-slave-nj21 krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060595, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:35.529229 11321 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:35.530450 11321 webserver.cc:460] Webserver started at http://127.10.225.190:41009/ using document root <none> and password file <none>
I20260806 23:56:35.531036 11321 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:35.531102 11321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:35.531301 11321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:35.533481 11321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "7499d24a6f114b4fac6b3a604594b2ed"
format_stamp: "Formatted at 2026-08-06 23:56:35 on dist-test-slave-nj21"
server_key: "54a4bdcf2cd5bfdcc54e1667b05c4c69"
server_key_iv: "90071d762f6b0969110133757a6e27d5"
server_key_version: "encryptionkey@0"
I20260806 23:56:35.534050 11321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "7499d24a6f114b4fac6b3a604594b2ed"
format_stamp: "Formatted at 2026-08-06 23:56:35 on dist-test-slave-nj21"
server_key: "54a4bdcf2cd5bfdcc54e1667b05c4c69"
server_key_iv: "90071d762f6b0969110133757a6e27d5"
server_key_version: "encryptionkey@0"
I20260806 23:56:35.537966 11321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260806 23:56:35.540587 11344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:35.541941 11321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 23:56:35.542064 11321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "7499d24a6f114b4fac6b3a604594b2ed"
format_stamp: "Formatted at 2026-08-06 23:56:35 on dist-test-slave-nj21"
server_key: "54a4bdcf2cd5bfdcc54e1667b05c4c69"
server_key_iv: "90071d762f6b0969110133757a6e27d5"
server_key_version: "encryptionkey@0"
I20260806 23:56:35.542176 11321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:35.557206 11321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:35.560361 11321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:35.560546 11321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:35.568001 11321 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:38383
I20260806 23:56:35.568091 11419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:38383 every 8 connection(s)
I20260806 23:56:35.568991 11321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:35.570487 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11321
I20260806 23:56:35.570641 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:35.570909 11142 external_mini_cluster.cc:1428] Setting key 7e8e97e506ff95f6ef643c4d9a766643
I20260806 23:56:35.572149 11421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:35.579035 11421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed: Bootstrap starting.
Aug 06 23:56:35 dist-test-slave-nj21 krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:35.582180 11421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:35.583061 11421 log.cc:824] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:35.585487 11421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed: No bootstrap required, opened a new log
I20260806 23:56:35.587167 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:35.572386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49847 (local address 127.10.225.190:38383)
0806 23:56:35.572834 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:35.572862 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:35.572908 (+ 46us) server_negotiation.cc:407] Connection header received
0806 23:56:35.573739 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:35.573761 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:35.574190 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:35.574552 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:35.575246 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:35.576760 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:35.577667 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:35.578368 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:35.578524 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:35.581782 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:35.581789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:35.581807 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:35.581843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:35.584023 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:35.584472 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:35.584478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:35.584484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:35.584621 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:35.584840 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:35.584843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:35.584845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:35.585374 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:35.585566 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:35.585909 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 23:56:35.586214 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":122,"threads_started":1}
I20260806 23:56:35.589984 11421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499d24a6f114b4fac6b3a604594b2ed" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38383 } }
I20260806 23:56:35.590241 11421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:35.590305 11421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7499d24a6f114b4fac6b3a604594b2ed, State: Initialized, Role: FOLLOWER
I20260806 23:56:35.590307 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:35.590723 11421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499d24a6f114b4fac6b3a604594b2ed" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38383 } }
I20260806 23:56:35.590857 11421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:35.590920 11421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:35.591013 11421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:35.592088 11421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499d24a6f114b4fac6b3a604594b2ed" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38383 } }
I20260806 23:56:35.592445 11421 leader_election.cc:302] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7499d24a6f114b4fac6b3a604594b2ed; no voters:
I20260806 23:56:35.592772 11421 leader_election.cc:288] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:35.592893 11440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:35.593144 11440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [term 1 LEADER]: Becoming Leader. State: Replica: 7499d24a6f114b4fac6b3a604594b2ed, State: Running, Role: LEADER
I20260806 23:56:35.593591 11440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499d24a6f114b4fac6b3a604594b2ed" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38383 } }
I20260806 23:56:35.594046 11421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:35.594801 11443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7499d24a6f114b4fac6b3a604594b2ed. Latest consensus state: current_term: 1 leader_uuid: "7499d24a6f114b4fac6b3a604594b2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499d24a6f114b4fac6b3a604594b2ed" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38383 } } }
I20260806 23:56:35.594925 11443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:35.595868 11441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7499d24a6f114b4fac6b3a604594b2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7499d24a6f114b4fac6b3a604594b2ed" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 38383 } } }
I20260806 23:56:35.595975 11441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed [sys.catalog]: This master's current role is: LEADER
W20260806 23:56:35.599393 11467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:35.599468 11467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:35.599879 11460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:35.600757 11460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:35.604099 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:35.592399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34765 (local address 127.10.225.190:38383)
0806 23:56:35.592516 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:35.592520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:35.592629 (+ 109us) server_negotiation.cc:407] Connection header received
0806 23:56:35.592736 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:35.592738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:35.592787 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:35.592866 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:35.593900 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:35.594709 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:35.596536 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:35.596864 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:35.596965 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:35.599112 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:35.599116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:35.599119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:35.599145 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:35.601183 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:35.601702 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:35.601706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:35.601708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:35.601805 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:35.602120 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:35.602123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:35.602125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:35.602543 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:35.602741 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:35.603776 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0806 23:56:35.603935 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:56:35.607297 11460 catalog_manager.cc:1347] Generated new cluster ID: e8eac8a08c244ae68f43efb7b1432c1f
I20260806 23:56:35.607374 11460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:35.630416 11460 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:35.632062 11460 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:35.644855 11460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7499d24a6f114b4fac6b3a604594b2ed: Generated new TSK 0
I20260806 23:56:35.645788 11460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:35.661553 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11321
2026-08-06T23:56:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1551 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:35 dist-test-slave-nj21 krb5kdc[11473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:35 dist-test-slave-nj21 krb5kdc[11473](info): set up 2 sockets
Aug 06 23:56:35 dist-test-slave-nj21 krb5kdc[11473](info): commencing operation
krb5kdc: starting...
W20260806 23:56:36.797840 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:36 dist-test-slave-nj21 krb5kdc[11473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:36.924908 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:43899
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:43899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:37.045945 11493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:37.046228 11493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:37.046272 11493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:37.050431 11493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:37.050511 11493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:37.050530 11493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:37.050545 11493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:37.050560 11493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:37.056058 11493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:43899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:37.057191 11493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:37.058590 11493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:37.066938 11500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:37.066922 11502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:37.066918 11504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:37.067327 11493 server_base.cc:1034] running on GCE node
I20260806 23:56:37.067958 11493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:37.069088 11493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:37.070286 11493 hybrid_clock.cc:648] HybridClock initialized: now 1786060597070245 us; error 51 us; skew 500 ppm
Aug 06 23:56:37 dist-test-slave-nj21 krb5kdc[11473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060597, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:37.073194 11493 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:37.074074 11493 webserver.cc:460] Webserver started at http://127.10.225.190:46809/ using document root <none> and password file <none>
I20260806 23:56:37.074620 11493 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:37.074676 11493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:37.074847 11493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:37.076941 11493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "8639d100c5b44b70a7485473ebb6ea82"
format_stamp: "Formatted at 2026-08-06 23:56:37 on dist-test-slave-nj21"
server_key: "357b7931c8bf6e2aa9793a5ce6e21fdc"
server_key_iv: "30a2594336df25670eef182adf378d3d"
server_key_version: "encryptionkey@0"
I20260806 23:56:37.077466 11493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "8639d100c5b44b70a7485473ebb6ea82"
format_stamp: "Formatted at 2026-08-06 23:56:37 on dist-test-slave-nj21"
server_key: "357b7931c8bf6e2aa9793a5ce6e21fdc"
server_key_iv: "30a2594336df25670eef182adf378d3d"
server_key_version: "encryptionkey@0"
I20260806 23:56:37.081187 11493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 23:56:37.083650 11516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:37.084913 11493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:37.085026 11493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "8639d100c5b44b70a7485473ebb6ea82"
format_stamp: "Formatted at 2026-08-06 23:56:37 on dist-test-slave-nj21"
server_key: "357b7931c8bf6e2aa9793a5ce6e21fdc"
server_key_iv: "30a2594336df25670eef182adf378d3d"
server_key_version: "encryptionkey@0"
I20260806 23:56:37.085129 11493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:37.109652 11493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:37.113552 11493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:37.113767 11493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:37.121161 11493 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:43899
I20260806 23:56:37.121415 11588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:43899 every 8 connection(s)
I20260806 23:56:37.122160 11493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:37.122877 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11493
I20260806 23:56:37.123008 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:37.123271 11142 external_mini_cluster.cc:1428] Setting key 1f51531be295440083531076ccc835f6
I20260806 23:56:37.125403 11589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:37.131870 11589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82: Bootstrap starting.
I20260806 23:56:37.133811 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:37.124643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36517 (local address 127.10.225.190:43899)
0806 23:56:37.125039 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:37.125056 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:37.125092 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:56:37.125770 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:37.125788 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:37.126234 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:37.126653 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:37.127011 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:37.128062 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:37.129360 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:37.130067 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:37.130198 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:37.131116 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:37.131121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:37.131134 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:37.131901 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:37.132006 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:37.132052 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:37.132179 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:37.132411 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 23:56:37.132773 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260806 23:56:37.134896 11589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:37.135900 11589 log.cc:824] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:37.137363 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:37.138124 11589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82: No bootstrap required, opened a new log
I20260806 23:56:37.142971 11589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8639d100c5b44b70a7485473ebb6ea82" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43899 } }
I20260806 23:56:37.143270 11589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:37.143347 11589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8639d100c5b44b70a7485473ebb6ea82, State: Initialized, Role: FOLLOWER
I20260806 23:56:37.143560 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:37.139376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47607 (local address 127.10.225.190:43899)
0806 23:56:37.139489 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:37.139493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:37.139944 (+ 451us) server_negotiation.cc:407] Connection header received
0806 23:56:37.140044 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:37.140048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:37.140094 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:37.140160 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:37.140422 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:37.140940 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:37.141727 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:37.141987 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:37.142075 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:37.143127 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:37.143130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:37.143132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:37.143143 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:37.143169 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:37.143179 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:37.143238 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:37.143360 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 23:56:37.143424 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 23:56:37.143867 11589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8639d100c5b44b70a7485473ebb6ea82" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43899 } }
I20260806 23:56:37.144045 11589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:37.144117 11589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:37.144218 11589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:37.145339 11589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8639d100c5b44b70a7485473ebb6ea82" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43899 } }
I20260806 23:56:37.145678 11589 leader_election.cc:302] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8639d100c5b44b70a7485473ebb6ea82; no voters:
I20260806 23:56:37.145999 11589 leader_election.cc:288] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:37.146134 11605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:37.146373 11605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [term 1 LEADER]: Becoming Leader. State: Replica: 8639d100c5b44b70a7485473ebb6ea82, State: Running, Role: LEADER
I20260806 23:56:37.146703 11605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8639d100c5b44b70a7485473ebb6ea82" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43899 } }
I20260806 23:56:37.147280 11589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:37.148281 11606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8639d100c5b44b70a7485473ebb6ea82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8639d100c5b44b70a7485473ebb6ea82" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43899 } } }
I20260806 23:56:37.148408 11606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:37.148850 11607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8639d100c5b44b70a7485473ebb6ea82. Latest consensus state: current_term: 1 leader_uuid: "8639d100c5b44b70a7485473ebb6ea82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8639d100c5b44b70a7485473ebb6ea82" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 43899 } } }
I20260806 23:56:37.148947 11607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:37.149199 11615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:37.152415 11615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:37.158499 11615 catalog_manager.cc:1347] Generated new cluster ID: 446dc6d799004b51af9b2f73fedbc524
I20260806 23:56:37.158571 11615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:37.166376 11615 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:37.167647 11615 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:37.176216 11615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8639d100c5b44b70a7485473ebb6ea82: Generated new TSK 0
I20260806 23:56:37.177085 11615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:37.186869 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11493
2026-08-06T23:56:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1527 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:37 dist-test-slave-nj21 krb5kdc[11632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:37 dist-test-slave-nj21 krb5kdc[11632](info): set up 2 sockets
Aug 06 23:56:37 dist-test-slave-nj21 krb5kdc[11632](info): commencing operation
krb5kdc: starting...
W20260806 23:56:38.323990 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:38 dist-test-slave-nj21 krb5kdc[11632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:38.453015 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:37317
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:38.567128 11653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:38.567548 11653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:38.567628 11653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:38.571704 11653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:38.571770 11653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:38.571789 11653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:38.571805 11653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:38.571837 11653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:38.577229 11653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:37317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:38.578361 11653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:38.579394 11653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:38.587808 11662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:38.587849 11653 server_base.cc:1034] running on GCE node
W20260806 23:56:38.587795 11664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:38.587795 11661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:38.588656 11653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:38.589838 11653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:38.591002 11653 hybrid_clock.cc:648] HybridClock initialized: now 1786060598590989 us; error 27 us; skew 500 ppm
Aug 06 23:56:38 dist-test-slave-nj21 krb5kdc[11632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060598, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:38.593720 11653 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:38.594633 11653 webserver.cc:460] Webserver started at http://127.10.225.190:41517/ using document root <none> and password file <none>
I20260806 23:56:38.595163 11653 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:38.595220 11653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:38.595397 11653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:38.597636 11653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "1c6af2795ba4448b8935bfac66034ea9"
format_stamp: "Formatted at 2026-08-06 23:56:38 on dist-test-slave-nj21"
server_key: "0b7b4eb9f7f8514d0e89fbdad0b8d542"
server_key_iv: "6d587f9198318736a961c89897854059"
server_key_version: "encryptionkey@0"
I20260806 23:56:38.598187 11653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "1c6af2795ba4448b8935bfac66034ea9"
format_stamp: "Formatted at 2026-08-06 23:56:38 on dist-test-slave-nj21"
server_key: "0b7b4eb9f7f8514d0e89fbdad0b8d542"
server_key_iv: "6d587f9198318736a961c89897854059"
server_key_version: "encryptionkey@0"
I20260806 23:56:38.602492 11653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:38.605248 11674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:38.606634 11653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 23:56:38.606767 11653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "1c6af2795ba4448b8935bfac66034ea9"
format_stamp: "Formatted at 2026-08-06 23:56:38 on dist-test-slave-nj21"
server_key: "0b7b4eb9f7f8514d0e89fbdad0b8d542"
server_key_iv: "6d587f9198318736a961c89897854059"
server_key_version: "encryptionkey@0"
I20260806 23:56:38.606887 11653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:38.612546 11653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:38.615585 11653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:38.615804 11653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:38.622777 11653 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:37317
I20260806 23:56:38.623098 11747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:37317 every 8 connection(s)
I20260806 23:56:38.623798 11653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:38.626675 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:38.630520 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11653
I20260806 23:56:38.630666 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:38.630956 11142 external_mini_cluster.cc:1428] Setting key 21516493ddd27b6724a3d1f0fa92ff68
I20260806 23:56:38.633028 11748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9: Bootstrap starting.
I20260806 23:56:38.635545 11748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:38.636265 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:38.632387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45887 (local address 127.10.225.190:37317)
0806 23:56:38.632792 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:38.632808 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:38.632843 (+ 35us) server_negotiation.cc:407] Connection header received
0806 23:56:38.633519 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:38.633537 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:38.633818 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:38.634093 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:38.634920 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:38.634925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:38.634948 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:38.634959 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:38.635022 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:38.635052 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:38.635144 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:38.635298 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 23:56:38.635466 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260806 23:56:38.636334 11748 log.cc:824] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:38.638441 11748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9: No bootstrap required, opened a new log
I20260806 23:56:38.639211 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:38.642912 11748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6af2795ba4448b8935bfac66034ea9" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37317 } }
I20260806 23:56:38.643172 11748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 23:56:38.643235 11748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6af2795ba4448b8935bfac66034ea9, State: Initialized, Role: FOLLOWER
I20260806 23:56:38.643739 11748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6af2795ba4448b8935bfac66034ea9" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37317 } }
I20260806 23:56:38.643967 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:38.641823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57749 (local address 127.10.225.190:37317)
0806 23:56:38.641953 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:38.641958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:38.642096 (+ 138us) server_negotiation.cc:407] Connection header received
0806 23:56:38.642192 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:38.642195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:38.642231 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:38.642322 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:38.643496 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:38.643500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:38.643502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 23:56:38.643514 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:38.643545 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 23:56:38.643557 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:38.643643 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:38.643808 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 23:56:38.643851 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 23:56:38.644078 11748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 23:56:38.644147 11748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 23:56:38.644251 11748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:38.645313 11748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6af2795ba4448b8935bfac66034ea9" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37317 } }
I20260806 23:56:38.645646 11748 leader_election.cc:302] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c6af2795ba4448b8935bfac66034ea9; no voters:
I20260806 23:56:38.645941 11748 leader_election.cc:288] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 23:56:38.646087 11759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:38.646348 11759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [term 1 LEADER]: Becoming Leader. State: Replica: 1c6af2795ba4448b8935bfac66034ea9, State: Running, Role: LEADER
I20260806 23:56:38.646659 11759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6af2795ba4448b8935bfac66034ea9" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37317 } }
I20260806 23:56:38.647164 11748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:38.648415 11762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c6af2795ba4448b8935bfac66034ea9. Latest consensus state: current_term: 1 leader_uuid: "1c6af2795ba4448b8935bfac66034ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6af2795ba4448b8935bfac66034ea9" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37317 } } }
I20260806 23:56:38.648422 11760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c6af2795ba4448b8935bfac66034ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6af2795ba4448b8935bfac66034ea9" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 37317 } } }
I20260806 23:56:38.648552 11762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:38.648566 11760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9 [sys.catalog]: This master's current role is: LEADER
W20260806 23:56:38.652486 11781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:38.652567 11781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:38.652750 11767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:38.654393 11767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:38.660683 11767 catalog_manager.cc:1347] Generated new cluster ID: 3399aef36fb9412e8151a3fc3aae7f9b
I20260806 23:56:38.660784 11767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:38.676918 11767 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:38.677263 11767 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:38.688241 11767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c6af2795ba4448b8935bfac66034ea9: Generated new TSK 0
I20260806 23:56:38.689000 11767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 23:56:38.695755 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11653
2026-08-06T23:56:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1507 ms)
[----------] 18 tests from AuthTokenIssuingTest (27804 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:38 dist-test-slave-nj21 krb5kdc[11789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:38 dist-test-slave-nj21 krb5kdc[11789](info): set up 2 sockets
Aug 06 23:56:38 dist-test-slave-nj21 krb5kdc[11789](info): commencing operation
krb5kdc: starting...
W20260806 23:56:39.838855 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:39 dist-test-slave-nj21 krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:39.980712 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35459
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35459,127.10.225.189:38237,127.10.225.188:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:40.095666 11804 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 23:56:40.095991 11804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:40.096015 11804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:40.096065 11804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:40.100117 11804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:40.100184 11804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:40.100203 11804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:40.100219 11804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:40.100234 11804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:40.105675 11804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35459,127.10.225.189:38237,127.10.225.188:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:40.106860 11804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:40.107942 11804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:40.116456 11812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:40.116420 11815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:40.116441 11810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:40.116696 11804 server_base.cc:1034] running on GCE node
I20260806 23:56:40.117305 11804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:40.118397 11804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:40.119551 11804 hybrid_clock.cc:648] HybridClock initialized: now 1786060600119540 us; error 28 us; skew 500 ppm
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:40.125365 11804 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:40.126317 11804 webserver.cc:460] Webserver started at http://127.10.225.190:39191/ using document root <none> and password file <none>
I20260806 23:56:40.126883 11804 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:40.126953 11804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:40.127166 11804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:40.129439 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "1a8813a23cf94d068a285bb41424dc70"
format_stamp: "Formatted at 2026-08-06 23:56:40 on dist-test-slave-nj21"
server_key: "2527c842fba12ff036f710315fd3bfba"
server_key_iv: "bbc39d5050212249c8ad633faccb3f6c"
server_key_version: "encryptionkey@0"
I20260806 23:56:40.130041 11804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "1a8813a23cf94d068a285bb41424dc70"
format_stamp: "Formatted at 2026-08-06 23:56:40 on dist-test-slave-nj21"
server_key: "2527c842fba12ff036f710315fd3bfba"
server_key_iv: "bbc39d5050212249c8ad633faccb3f6c"
server_key_version: "encryptionkey@0"
I20260806 23:56:40.133939 11804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 23:56:40.136499 11824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:40.137697 11804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:56:40.137823 11804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "1a8813a23cf94d068a285bb41424dc70"
format_stamp: "Formatted at 2026-08-06 23:56:40 on dist-test-slave-nj21"
server_key: "2527c842fba12ff036f710315fd3bfba"
server_key_iv: "bbc39d5050212249c8ad633faccb3f6c"
server_key_version: "encryptionkey@0"
I20260806 23:56:40.137928 11804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:40.152143 11804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:40.155150 11804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:40.155352 11804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:40.162617 11804 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:35459
I20260806 23:56:40.162890 11899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:35459 every 8 connection(s)
I20260806 23:56:40.163657 11804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:40.166582 11900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:40.167964 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11804
I20260806 23:56:40.168088 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:40.168346 11142 external_mini_cluster.cc:1428] Setting key 0f0de268d18b05da1cdd3a1b75f99590
I20260806 23:56:40.170717 11900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.189@KRBTEST.COM" created.
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:40.186617 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.173552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.10.225.190:35459)
0806 23:56:40.174082 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.174102 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.174361 (+ 259us) server_negotiation.cc:407] Connection header received
0806 23:56:40.175416 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.175422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.175657 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.175884 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.176676 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.178009 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.178970 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.179345 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.179477 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.182019 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.182026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.182041 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.182069 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.183643 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.184114 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.184118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.184119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.184223 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.184497 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.184500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.184502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.184884 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.185004 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.185352 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.185566 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":358,"thread_start_us":151,"threads_started":1}
I20260806 23:56:40.186617 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.173552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35459 (local address 127.0.0.1:49230)
0806 23:56:40.174200 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.174247 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:40.175100 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:40.175902 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:40.175906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:40.176224 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:40.176523 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.176537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.178125 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.178128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.178871 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.178881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.179476 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.179478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.179555 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.180155 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:40.180176 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:40.181861 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:40.183755 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.183763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.183772 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.183997 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.184308 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.184311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.184313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.184416 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.184990 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:40.184996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:40.185120 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 23:56:40.185267 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 23:56:40.185625 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":114,"spinlock_wait_cycles":55552,"thread_start_us":173,"threads_started":1}
I20260806 23:56:40.189108 11900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:40.189956 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.189612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:38237 (local address 127.0.0.1:49254)
0806 23:56:40.189790 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.189882 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 23:56:40.190188 11829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
I20260806 23:56:40.191049 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.190851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:38237 (local address 127.0.0.1:49266)
0806 23:56:40.190946 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.190994 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 23:56:40.191187 11900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.189:38237: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab.
Entry for principal kudu/127.10.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab.
I20260806 23:56:40.196269 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.189:38237
--webserver_interface=127.10.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab
--principal=kudu/127.10.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35459,127.10.225.189:38237,127.10.225.188:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:56:40.240451 11900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } attempt: 1
I20260806 23:56:40.241416 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.241039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:38237 (local address 127.0.0.1:49280)
0806 23:56:40.241228 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.241319 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 23:56:40.242162 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.241927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:38237 (local address 127.0.0.1:49292)
0806 23:56:40.242028 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.242101 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 23:56:40.242331 11900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.189:38237: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
W20260806 23:56:40.309783 11911 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 23:56:40.310118 11911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:40.310153 11911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:40.310211 11911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 23:56:40.310518 11900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } attempt: 2
I20260806 23:56:40.311401 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.311040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:38237 (local address 127.0.0.1:49302)
0806 23:56:40.311223 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.311299 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 23:56:40.312189 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.311980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:38237 (local address 127.0.0.1:49304)
0806 23:56:40.312076 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.312126 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 23:56:40.312336 11900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.189:38237: Network error: Client connection negotiation failed: client connection to 127.10.225.189:38237: connect: Connection refused (error 111)
W20260806 23:56:40.314729 11911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:40.314786 11911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:40.314805 11911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:40.314821 11911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:40.314836 11911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:40.320294 11911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35459,127.10.225.189:38237,127.10.225.188:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab
--principal=kudu/127.10.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.189:38237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:40.321337 11911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:40.322316 11911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:40.330696 11918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:40.330765 11922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:40.330766 11917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:40.331095 11911 server_base.cc:1034] running on GCE node
I20260806 23:56:40.331631 11911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:40.332762 11911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:40.333971 11911 hybrid_clock.cc:648] HybridClock initialized: now 1786060600333935 us; error 46 us; skew 500 ppm
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:40.336843 11911 init.cc:377] Logged in from keytab as kudu/127.10.225.189@KRBTEST.COM (short username kudu)
I20260806 23:56:40.337944 11911 webserver.cc:460] Webserver started at http://127.10.225.189:36411/ using document root <none> and password file <none>
I20260806 23:56:40.338512 11911 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:40.338579 11911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:40.338797 11911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:40.340953 11911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/data/instance:
uuid: "6ff6c53966f542c0aef50517c4bc8525"
format_stamp: "Formatted at 2026-08-06 23:56:40 on dist-test-slave-nj21"
server_key: "04af4c3af41f3249c25491492abe630f"
server_key_iv: "e9e037702124d7afd48b0e0dc027d56b"
server_key_version: "encryptionkey@0"
I20260806 23:56:40.341502 11911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/wal/instance:
uuid: "6ff6c53966f542c0aef50517c4bc8525"
format_stamp: "Formatted at 2026-08-06 23:56:40 on dist-test-slave-nj21"
server_key: "04af4c3af41f3249c25491492abe630f"
server_key_iv: "e9e037702124d7afd48b0e0dc027d56b"
server_key_version: "encryptionkey@0"
I20260806 23:56:40.345441 11911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 23:56:40.347791 11936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:40.348964 11911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 23:56:40.349097 11911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/wal
uuid: "6ff6c53966f542c0aef50517c4bc8525"
format_stamp: "Formatted at 2026-08-06 23:56:40 on dist-test-slave-nj21"
server_key: "04af4c3af41f3249c25491492abe630f"
server_key_iv: "e9e037702124d7afd48b0e0dc027d56b"
server_key_version: "encryptionkey@0"
I20260806 23:56:40.349196 11911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:40.365075 11911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:40.368063 11911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:40.368242 11911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:40.375136 11911 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.189:38237
I20260806 23:56:40.375396 12005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.189:38237 every 8 connection(s)
I20260806 23:56:40.376161 11911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
I20260806 23:56:40.378943 12006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:40.382797 12006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:40.383872 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 11911
I20260806 23:56:40.383999 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-1/wal/instance
I20260806 23:56:40.384249 11142 external_mini_cluster.cc:1428] Setting key 2e856610de351863e87ebb6300944925
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.188@KRBTEST.COM" created.
I20260806 23:56:40.397518 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.384982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.10.225.190:35459)
0806 23:56:40.385118 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.385121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.386150 (+ 1029us) server_negotiation.cc:407] Connection header received
0806 23:56:40.387369 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.387373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.387426 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.387504 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.388783 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.389617 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.390774 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.391140 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.391252 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.393805 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.393808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.393810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.393838 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.395271 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.395898 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.395902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.395904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.395998 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.396359 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.396362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.396364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.396592 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.396685 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.397216 (+ 531us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.397350 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 23:56:40.398301 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.385447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35459 (local address 127.0.0.1:49236)
0806 23:56:40.385995 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.386037 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:40.387110 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:40.387724 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:40.387733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:40.388126 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:40.388626 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.388638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.389750 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.389753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.390639 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.390649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.391307 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.391310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.391457 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.392031 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:40.392048 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:40.393605 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:40.395380 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.395386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.395403 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.395780 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.396094 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.396097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.396099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.396260 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.396688 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:40.396693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:40.396963 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 23:56:40.397118 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 23:56:40.397388 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":162,"threads_started":1}
I20260806 23:56:40.399858 12006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:40.403517 11900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } attempt: 3
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
I20260806 23:56:40.411577 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.400385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:38237 (local address 127.0.0.1:49318)
0806 23:56:40.400558 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.400571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:40.400649 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:40.401306 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:40.401308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:40.401459 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:40.401551 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.401556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.402940 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.402942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.403491 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.403499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.404516 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.404520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.404606 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.405582 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:40.405595 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:40.407847 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:40.410034 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.410037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.410039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.410214 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.410513 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.410516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.410518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.410594 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.411109 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:40.411112 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:40.411201 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 23:56:40.411304 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 23:56:40.411444 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 23:56:40.411803 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.400495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.10.225.189:38237)
0806 23:56:40.400871 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.400882 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.400918 (+ 36us) server_negotiation.cc:407] Connection header received
0806 23:56:40.400964 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.400970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.401182 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.401327 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.401654 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.402812 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.403598 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.404170 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.404269 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.407987 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.407997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.408010 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.408058 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.409944 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.410320 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.410326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.410328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.410422 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.410672 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.410675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.410677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.411012 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.411139 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.411428 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.411593 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":159,"threads_started":1}
I20260806 23:56:40.413426 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.404226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.10.225.189:38237)
0806 23:56:40.404674 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.404683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.404697 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:56:40.404733 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.404735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.404783 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.404877 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.405699 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.406393 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.407232 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.407461 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.407577 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.409686 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.409689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.409691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.409714 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.411962 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.412324 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.412328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.412329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.412413 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.412636 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.412639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.412641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.412842 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.412924 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.413202 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.413293 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":214,"threads_started":1}
I20260806 23:56:40.413465 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.404034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:38237 (local address 127.0.0.1:49334)
0806 23:56:40.404180 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.404196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:40.404284 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:40.404924 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:40.404927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:40.405104 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:40.405576 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.405585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.406494 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.406498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.407106 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.407117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.407585 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.407589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.407735 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.408170 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:40.408184 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:40.409529 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:40.412051 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.412054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.412055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.412242 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.412498 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.412500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.412501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.412566 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.412937 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:40.412940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:40.413047 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 23:56:40.413160 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 23:56:40.413297 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 23:56:40.413892 12006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab.
Entry for principal kudu/127.10.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab.
I20260806 23:56:40.414255 11900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:40.414647 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.414392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49196)
0806 23:56:40.414523 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.414596 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 23:56:40.414844 11940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
I20260806 23:56:40.415248 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.415023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49198)
0806 23:56:40.415124 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.415176 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 23:56:40.415738 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.188:33897
--webserver_interface=127.10.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab
--principal=kudu/127.10.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:35459,127.10.225.189:38237,127.10.225.188:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:56:40.415885 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.415628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49204)
0806 23:56:40.415782 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.415828 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 23:56:40.415928 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.415648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49218)
0806 23:56:40.415811 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.415859 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260806 23:56:40.415988 11900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:33897: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
W20260806 23:56:40.417400 12006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:33897: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
I20260806 23:56:40.447153 11900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } attempt: 1
I20260806 23:56:40.448145 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.447792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49234)
0806 23:56:40.447968 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.448045 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 23:56:40.449025 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.448731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49248)
0806 23:56:40.448878 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.448952 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 23:56:40.449170 11900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:33897: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
I20260806 23:56:40.466657 12006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } attempt: 1
I20260806 23:56:40.467614 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.467274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49258)
0806 23:56:40.467462 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.467526 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 23:56:40.468314 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.468111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49260)
0806 23:56:40.468213 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.468257 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 23:56:40.468462 12006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:33897: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
I20260806 23:56:40.524366 11900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } attempt: 2
I20260806 23:56:40.525249 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.524918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49268)
0806 23:56:40.525086 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.525157 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 23:56:40.526306 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.526024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49276)
0806 23:56:40.526175 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.526233 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 23:56:40.526479 11900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:33897: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
W20260806 23:56:40.534826 12017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 23:56:40.535109 12017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:40.535156 12017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:40.535220 12017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 23:56:40.536613 12006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } attempt: 2
I20260806 23:56:40.537511 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.537142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49292)
0806 23:56:40.537362 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.537431 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 23:56:40.538277 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.537958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49296)
0806 23:56:40.538152 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.538208 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260806 23:56:40.538405 12006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:33897: Network error: Client connection negotiation failed: client connection to 127.10.225.188:33897: connect: Connection refused (error 111)
W20260806 23:56:40.539943 12017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:40.540002 12017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:40.540021 12017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:40.540037 12017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:40.540050 12017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:40.546092 12017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:35459,127.10.225.189:38237,127.10.225.188:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab
--principal=kudu/127.10.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.188:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:40.547499 12017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:40.548802 12017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:40.557801 12025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:40.557811 12028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:40.557837 12026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:40.558115 12017 server_base.cc:1034] running on GCE node
I20260806 23:56:40.558674 12017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:40.559847 12017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:40.561031 12017 hybrid_clock.cc:648] HybridClock initialized: now 1786060600561003 us; error 38 us; skew 500 ppm
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:40.563913 12017 init.cc:377] Logged in from keytab as kudu/127.10.225.188@KRBTEST.COM (short username kudu)
I20260806 23:56:40.564824 12017 webserver.cc:460] Webserver started at http://127.10.225.188:40605/ using document root <none> and password file <none>
I20260806 23:56:40.565367 12017 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:40.565429 12017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:40.565732 12017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:40.568048 12017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/data/instance:
uuid: "99dd6e466b3346029e0d2e20697bbdbd"
format_stamp: "Formatted at 2026-08-06 23:56:40 on dist-test-slave-nj21"
server_key: "1f9535c325f5ac1e952f2ab45f1b186a"
server_key_iv: "aa658e34c59c0b9d4cbcc76804927036"
server_key_version: "encryptionkey@0"
I20260806 23:56:40.568607 12017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/wal/instance:
uuid: "99dd6e466b3346029e0d2e20697bbdbd"
format_stamp: "Formatted at 2026-08-06 23:56:40 on dist-test-slave-nj21"
server_key: "1f9535c325f5ac1e952f2ab45f1b186a"
server_key_iv: "aa658e34c59c0b9d4cbcc76804927036"
server_key_version: "encryptionkey@0"
I20260806 23:56:40.572511 12017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:56:40.574977 12038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:40.576383 12017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 23:56:40.576503 12017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/wal
uuid: "99dd6e466b3346029e0d2e20697bbdbd"
format_stamp: "Formatted at 2026-08-06 23:56:40 on dist-test-slave-nj21"
server_key: "1f9535c325f5ac1e952f2ab45f1b186a"
server_key_iv: "aa658e34c59c0b9d4cbcc76804927036"
server_key_version: "encryptionkey@0"
I20260806 23:56:40.576622 12017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:40.604123 12017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:40.607354 12017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:40.607559 12017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:40.615103 12017 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.188:33897
I20260806 23:56:40.615911 12119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.188:33897 every 8 connection(s)
I20260806 23:56:40.616318 12017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
I20260806 23:56:40.619481 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:40.623435 12120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:40.623787 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 12017
I20260806 23:56:40.623931 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786060299689793-11142-0/minicluster-data/master-2/wal/instance
I20260806 23:56:40.624171 11142 external_mini_cluster.cc:1428] Setting key 35bf1fe90fdf8634bf05009e75313240
I20260806 23:56:40.625656 11900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } attempt: 3
I20260806 23:56:40.629554 12006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } attempt: 3
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
I20260806 23:56:40.638497 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.626260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.10.225.190:35459)
0806 23:56:40.626400 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.626405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.627096 (+ 691us) server_negotiation.cc:407] Connection header received
0806 23:56:40.627388 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.627391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.627449 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.627517 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.628511 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.628988 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.630306 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.630598 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.630673 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.634074 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.634078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.634081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.634109 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.635656 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.636485 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.636489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.636491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.636571 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.636979 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.636983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.636984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.637277 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.637356 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.638095 (+ 739us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.638228 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:56:40.639791 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.626114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49318)
0806 23:56:40.626486 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.626499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:40.626570 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:40.627494 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:40.627497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:40.627740 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:40.627846 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.627853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.629436 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.629439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.630228 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.630238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.631088 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.631092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.631179 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.631712 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:40.631726 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:40.633962 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:40.638178 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.638181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.638183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.638426 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.638705 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.638708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.638709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.638792 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.639316 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:40.639319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:40.639412 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 23:56:40.639490 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 23:56:40.639616 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320}
I20260806 23:56:40.639883 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.630014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49330)
0806 23:56:40.630208 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.630222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:40.630304 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:40.630551 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:40.630554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:40.630742 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:40.630850 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.630857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.631947 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.631951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.632726 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.632738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.633347 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.633352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.633444 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.634132 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:40.634147 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:40.635438 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:40.638062 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.638065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.638067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.638313 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.638910 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.638914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.638915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.639003 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.639386 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:40.639389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:40.639499 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 23:56:40.639574 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 23:56:40.639741 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260806 23:56:40.640583 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.626217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.10.225.188:33897)
0806 23:56:40.626723 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.626729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.626739 (+ 10us) server_negotiation.cc:407] Connection header received
0806 23:56:40.626896 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.626900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.627219 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.627455 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.627963 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.629322 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.630398 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.630957 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.631404 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.634416 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.634423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.634439 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.634465 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.638071 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.638531 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.638533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.638535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.638608 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.638897 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.638900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.638902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.639202 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.639974 (+ 772us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.640058 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.640298 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":418,"server-negotiator.queue_time_us":451,"spinlock_wait_cycles":5504,"thread_start_us":141,"threads_started":1}
I20260806 23:56:40.640622 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.626124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:35459 (local address 127.0.0.1:49242)
0806 23:56:40.626950 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.626999 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:40.627214 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:40.627735 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:40.627741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:40.628176 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:40.628377 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.628391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.629086 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.629089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.630168 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.630180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.630872 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.630876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.631044 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.631866 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:40.631886 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:40.633925 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:40.635816 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.635839 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.635852 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.636390 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.636657 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.636660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.636662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.636861 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.637443 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:40.637450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:40.637784 (+ 334us) client_negotiation.cc:769] Sending connection context
0806 23:56:40.638742 (+ 958us) client_negotiation.cc:240] Negotiation successful
0806 23:56:40.639108 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":89,"threads_started":1}
I20260806 23:56:40.641157 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.630032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.10.225.188:33897)
0806 23:56:40.630312 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.630321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.630344 (+ 23us) server_negotiation.cc:407] Connection header received
0806 23:56:40.630375 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.630377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.630424 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.630961 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.631095 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.631815 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.632878 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.633206 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.633313 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.635567 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.635570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.635572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.635596 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.637932 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.638469 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.638474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.638476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.638808 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.639116 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.639119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.639121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.639289 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.640485 (+ 1196us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.640558 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.641009 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260806 23:56:40.640583 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.625459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.10.225.188:33897)
0806 23:56:40.625884 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.625902 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.625952 (+ 50us) server_negotiation.cc:407] Connection header received
0806 23:56:40.626881 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.626900 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.627218 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.627963 (+ 745us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.628387 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.629329 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.631351 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.631754 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.631849 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.634539 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.634543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.634545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.634566 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.637991 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.638593 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.638597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.638598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.638703 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.638999 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.639003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.639004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.639203 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.639385 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.640041 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.640210 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":293,"server-negotiator.queue_time_us":284,"spinlock_wait_cycles":16000,"thread_start_us":122,"threads_started":1}
I20260806 23:56:40.643025 12120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:40.645119 11900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70: Bootstrap starting.
I20260806 23:56:40.645146 12006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525: Bootstrap starting.
I20260806 23:56:40.647550 12006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:40.647646 11900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:40.648301 12006 log.cc:824] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:40.648540 11900 log.cc:824] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70: Log is configured to *not* fsync() on all Append() calls
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
I20260806 23:56:40.650612 12006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525: No bootstrap required, opened a new log
I20260806 23:56:40.652567 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.643656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.10.225.189:38237)
0806 23:56:40.643800 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.643804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.643837 (+ 33us) server_negotiation.cc:407] Connection header received
0806 23:56:40.643935 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.643937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.643994 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.644072 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.644513 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.645258 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.646298 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.646641 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.646746 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.649222 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.649226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.649229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.649252 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.650765 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.651175 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.651179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.651181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.651268 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.651536 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.651539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.651541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.651859 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.651946 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.652283 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.652406 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 23:56:40.653101 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.643565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:38237 (local address 127.0.0.1:49344)
0806 23:56:40.643771 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.643785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:40.643855 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:40.644096 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:40.644099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:40.644272 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:40.644390 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.644399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.645389 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.645392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.646162 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.646175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.646763 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.646767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.646864 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.647379 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:40.647393 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:40.649070 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:40.650877 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.650880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.650882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.651080 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.651367 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.651369 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.651370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.651451 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.652006 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:40.652009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:40.652111 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 23:56:40.652785 (+ 674us) client_negotiation.cc:240] Negotiation successful
0806 23:56:40.652942 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260806 23:56:40.653906 12120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:40.654562 11900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70: No bootstrap required, opened a new log
I20260806 23:56:40.655519 12006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } }
I20260806 23:56:40.655789 12006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff6c53966f542c0aef50517c4bc8525, State: Initialized, Role: FOLLOWER
I20260806 23:56:40.656154 12006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } }
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:40.657732 12006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:40.659958 11900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } }
I20260806 23:56:40.660185 11900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8813a23cf94d068a285bb41424dc70, State: Initialized, Role: FOLLOWER
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
I20260806 23:56:40.662096 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.649477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.10.225.190:35459)
0806 23:56:40.649593 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.649598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.651919 (+ 2321us) server_negotiation.cc:407] Connection header received
0806 23:56:40.652031 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.652034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.652082 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.652157 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.652877 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.653574 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.654627 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.654929 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.655030 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.657906 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.657910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.657912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.657942 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.659742 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.660263 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.660267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.660269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.660360 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.660666 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.660670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.660672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.660911 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.661092 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.661814 (+ 722us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.661935 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 23:56:40.664178 12147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:40.664258 12147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:40.664558 12134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.664701 12134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:40.664981 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.654465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.10.225.188:33897)
0806 23:56:40.654567 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.654572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.654678 (+ 106us) server_negotiation.cc:407] Connection header received
0806 23:56:40.654798 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.654801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.654832 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.654871 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.655726 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.656425 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.657955 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.658684 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.658761 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.661413 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.661416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.661419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.661440 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.663318 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.663765 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.663768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.663770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.663844 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.664070 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.664073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.664074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.664250 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.664341 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.664699 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.664810 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 23:56:40.662021 11911 master_runner.cc:186] Error getting master registration for 127.10.225.190:35459: OK, instance_id { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" instance_seqno: 1786060600160549 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 23:56:40.665354 11911 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:56:40.665452 11900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } }
I20260806 23:56:40.665645 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.654382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:33897 (local address 127.0.0.1:49338)
0806 23:56:40.654620 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:40.654633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:40.654726 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:40.654992 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:40.654995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:40.655217 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:40.655586 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.655597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.656713 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.656716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.657770 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:40.657783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.659133 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.659136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:40.659227 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.659652 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:40.659666 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:40.661290 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:40.663432 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.663435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.663436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.663653 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.663929 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:40.663932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:40.663933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:40.663998 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:40.664465 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:40.664467 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:40.664554 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 23:56:40.665379 (+ 825us) client_negotiation.cc:240] Negotiation successful
0806 23:56:40.665518 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260806 23:56:40.667053 11900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 23:56:40 dist-test-slave-nj21 krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
I20260806 23:56:40.669255 12120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd: Bootstrap starting.
I20260806 23:56:40.672152 12120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:40.673096 12120 log.cc:824] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:40.673352 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.663299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.10.225.189:38237)
0806 23:56:40.663399 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.663404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.663662 (+ 258us) server_negotiation.cc:407] Connection header received
0806 23:56:40.663734 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.663739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.663785 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.663847 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.664651 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.665254 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.666240 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.666599 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.666707 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.669730 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.669735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.669737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.669764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.671348 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.671855 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.671859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.671861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.671958 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.672267 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.672271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.672273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.672507 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.672701 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.673080 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.673204 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 23:56:40.677752 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 23:56:40.678764 12169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:40.678848 12169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:40.679010 11863 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1a8813a23cf94d068a285bb41424dc70"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49256
I20260806 23:56:40.679101 12149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.679224 12149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:40.679335 11863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 23:56:40.679461 11863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:40.675127 11804 master_runner.cc:186] Error getting master registration for 127.10.225.188:33897: OK, instance_id { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" instance_seqno: 1786060600612894 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 23:56:40.679863 11804 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:56:40.680867 11863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } }
I20260806 23:56:40.681931 11863 leader_election.cc:288] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [CANDIDATE]: Term 1 election: Requested vote from peers 6ff6c53966f542c0aef50517c4bc8525 (127.10.225.189:38237), 99dd6e466b3346029e0d2e20697bbdbd (127.10.225.188:33897)
I20260806 23:56:40.682360 11971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a8813a23cf94d068a285bb41424dc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ff6c53966f542c0aef50517c4bc8525"
I20260806 23:56:40.682576 11971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:40.684207 11971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a8813a23cf94d068a285bb41424dc70 in term 1.
I20260806 23:56:40.684645 11829 leader_election.cc:302] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a8813a23cf94d068a285bb41424dc70, 6ff6c53966f542c0aef50517c4bc8525; no voters:
I20260806 23:56:40.684942 12149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:40.685148 12149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [term 1 LEADER]: Becoming Leader. State: Replica: 1a8813a23cf94d068a285bb41424dc70, State: Running, Role: LEADER
I20260806 23:56:40.685516 12149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } }
I20260806 23:56:40.687182 12149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8813a23cf94d068a285bb41424dc70. Latest consensus state: current_term: 1 leader_uuid: "1a8813a23cf94d068a285bb41424dc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.687273 12149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:40.688709 12120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd: No bootstrap required, opened a new log
I20260806 23:56:40.692442 12193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:40.692531 12120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } }
I20260806 23:56:40.692790 12120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99dd6e466b3346029e0d2e20697bbdbd, State: Initialized, Role: FOLLOWER
I20260806 23:56:40.693118 12120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } }
I20260806 23:56:40.694594 12120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:40.694667 12084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a8813a23cf94d068a285bb41424dc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99dd6e466b3346029e0d2e20697bbdbd"
I20260806 23:56:40.694880 12084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:40.697099 12084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a8813a23cf94d068a285bb41424dc70 in term 1.
I20260806 23:56:40.702198 12193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:40.703927 12194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.704115 12194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:40.710331 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.684424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.10.225.189:38237)
0806 23:56:40.684693 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.684699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.694394 (+ 9695us) server_negotiation.cc:407] Connection header received
0806 23:56:40.699813 (+ 5419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.699817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.700056 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.700167 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.701493 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.702818 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.704353 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.704662 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.704749 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.706724 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.706728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.706730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.706757 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.708402 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.708899 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.708903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.708905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.709014 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.709318 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.709321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.709323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.709567 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.709671 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.710041 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.710171 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
W20260806 23:56:40.712883 12208 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:40.714447 12208 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:40.718012 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.685262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.10.225.190:35459)
0806 23:56:40.685363 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.685366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.702660 (+ 17294us) server_negotiation.cc:407] Connection header received
0806 23:56:40.705764 (+ 3104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.705768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.705828 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.705880 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.709329 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.710049 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.711319 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.711655 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.711764 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.714112 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.714116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.714119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.714150 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.715773 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.716285 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.716289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.716291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.716388 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.716710 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.716714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.716715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.716944 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.717055 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.717689 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.717839 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 23:56:40.720871 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:40.685841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.10.225.188:33897)
0806 23:56:40.685963 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.685967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.710973 (+ 25006us) server_negotiation.cc:407] Connection header received
0806 23:56:40.711132 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.711135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.711183 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.711232 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.712093 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.713024 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.714301 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:40.714853 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:40.715153 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:40.716751 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:40.716754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:40.716757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:40.716783 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:40.718988 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.719480 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.719484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.719485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.719576 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:40.719935 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:40.719938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:40.719940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:40.720158 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:40.720267 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:40.720594 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 23:56:40.720711 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 23:56:40.730166 12084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [term 1 FOLLOWER]: Refusing update from remote peer 1a8813a23cf94d068a285bb41424dc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:40.730319 11971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [term 1 FOLLOWER]: Refusing update from remote peer 1a8813a23cf94d068a285bb41424dc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:40.730945 12149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:40.731185 12184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:40.735133 12194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8813a23cf94d068a285bb41424dc70. Latest consensus state: current_term: 1 leader_uuid: "1a8813a23cf94d068a285bb41424dc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.735134 12134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8813a23cf94d068a285bb41424dc70. Latest consensus state: current_term: 1 leader_uuid: "1a8813a23cf94d068a285bb41424dc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.735255 12134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:40.735255 12194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:40.746543 12184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a8813a23cf94d068a285bb41424dc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.746709 12184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:40.746836 12210 mvcc.cc:205] Tried to move back new op lower bound from 7315704220561547264 to 7315704220411596800. Current Snapshot: MvccSnapshot[applied={T|T < 7315704220561547264}]
I20260806 23:56:40.748380 12194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a8813a23cf94d068a285bb41424dc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.748515 12194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:40.748605 12213 mvcc.cc:205] Tried to move back new op lower bound from 7315704220561547264 to 7315704220411596800. Current Snapshot: MvccSnapshot[applied={T|T < 7315704220561547264}]
I20260806 23:56:40.750072 12149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a8813a23cf94d068a285bb41424dc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.750203 12149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:40.751660 12134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a8813a23cf94d068a285bb41424dc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a8813a23cf94d068a285bb41424dc70" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 35459 } } peers { permanent_uuid: "6ff6c53966f542c0aef50517c4bc8525" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 38237 } } peers { permanent_uuid: "99dd6e466b3346029e0d2e20697bbdbd" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 33897 } } }
I20260806 23:56:40.751813 12134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:40.752318 12212 mvcc.cc:205] Tried to move back new op lower bound from 7315704220561547264 to 7315704220411596800. Current Snapshot: MvccSnapshot[applied={T|T < 7315704220561547264}]
I20260806 23:56:40.755460 12193 catalog_manager.cc:1347] Generated new cluster ID: 2a828ab342e346e2a5d5adf6e3b4bd84
I20260806 23:56:40.755532 12193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:40.779619 12193 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:40.781152 12193 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:40.797147 12193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a8813a23cf94d068a285bb41424dc70: Generated new TSK 0
I20260806 23:56:40.798105 12193 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 23:56:40.832461 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.829222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.10.225.189:38237)
0806 23:56:40.829356 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.829360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.831254 (+ 1894us) server_negotiation.cc:407] Connection header received
0806 23:56:40.831470 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.831474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.831530 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.831584 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.832299 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49366: BlockingRecv error: recv got EOF from 127.0.0.1:49366 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 23:56:40.834453 12126 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.830679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.10.225.188:33897)
0806 23:56:40.830788 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.830793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.833176 (+ 2383us) server_negotiation.cc:407] Connection header received
0806 23:56:40.833415 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.833418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.833493 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.833546 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.834308 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49350: BlockingRecv error: recv got EOF from 127.0.0.1:49350 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260806 23:56:40.835258 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.830006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.10.225.190:35459)
0806 23:56:40.830123 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.830128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.833002 (+ 2874us) server_negotiation.cc:407] Connection header received
0806 23:56:40.833686 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.833689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.833742 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.833789 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.835131 (+ 1342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49278: BlockingRecv error: recv got EOF from 127.0.0.1:49278 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 23:56:40.841166 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.839385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.10.225.189:38237)
0806 23:56:40.839522 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.839526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.840081 (+ 555us) server_negotiation.cc:407] Connection header received
0806 23:56:40.840307 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.840311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.840405 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.840460 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.841044 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49378: BlockingRecv error: recv got EOF from 127.0.0.1:49378 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 23:56:40.847299 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.845618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.10.225.189:38237)
0806 23:56:40.845734 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.845739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.846324 (+ 585us) server_negotiation.cc:407] Connection header received
0806 23:56:40.846518 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.846522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.846575 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.846626 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.847176 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49394: BlockingRecv error: recv got EOF from 127.0.0.1:49394 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 23:56:40.853197 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.851479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.10.225.189:38237)
0806 23:56:40.851589 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.851592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.852148 (+ 556us) server_negotiation.cc:407] Connection header received
0806 23:56:40.852329 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.852332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.852383 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.852430 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.853071 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49396: BlockingRecv error: recv got EOF from 127.0.0.1:49396 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 23:56:40.863404 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.860898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.10.225.189:38237)
0806 23:56:40.861110 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.861116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.861889 (+ 773us) server_negotiation.cc:407] Connection header received
0806 23:56:40.862150 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.862154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.862227 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.862289 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.863258 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49412: BlockingRecv error: recv got EOF from 127.0.0.1:49412 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260806 23:56:40.876560 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.874608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.10.225.189:38237)
0806 23:56:40.874746 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.874751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.875403 (+ 652us) server_negotiation.cc:407] Connection header received
0806 23:56:40.875632 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.875635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.875731 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.875790 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.876424 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49414: BlockingRecv error: recv got EOF from 127.0.0.1:49414 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 23:56:40.896811 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.895451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.10.225.189:38237)
0806 23:56:40.895585 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.895590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.895896 (+ 306us) server_negotiation.cc:407] Connection header received
0806 23:56:40.896027 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.896030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.896092 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.896179 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.896681 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49420: BlockingRecv error: recv got EOF from 127.0.0.1:49420 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 23:56:40.931977 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.930860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.10.225.189:38237)
0806 23:56:40.931023 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.931027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.931088 (+ 61us) server_negotiation.cc:407] Connection header received
0806 23:56:40.931201 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.931204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.931255 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.931348 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.931859 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49428: BlockingRecv error: recv got EOF from 127.0.0.1:49428 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 23:56:41.000052 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:40.998597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.10.225.189:38237)
0806 23:56:40.998740 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:40.998744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:40.998827 (+ 83us) server_negotiation.cc:407] Connection header received
0806 23:56:40.998970 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:40.998972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:40.999025 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:40.999095 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:40.999946 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49440: BlockingRecv error: recv got EOF from 127.0.0.1:49440 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 23:56:41.131625 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:41.130451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.10.225.189:38237)
0806 23:56:41.130579 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:41.130583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:41.130818 (+ 235us) server_negotiation.cc:407] Connection header received
0806 23:56:41.130924 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:41.130926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:41.130978 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:41.131047 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:41.131496 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49452: BlockingRecv error: recv got EOF from 127.0.0.1:49452 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 23:56:41.390544 12013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:41.389569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.10.225.189:38237)
0806 23:56:41.389710 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:41.389714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:41.389808 (+ 94us) server_negotiation.cc:407] Connection header received
0806 23:56:41.389937 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:41.389940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:41.389986 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:41.390055 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:41.390429 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49454: BlockingRecv error: recv got EOF from 127.0.0.1:49454 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:56:41.665607 12147 catalog_manager.cc:1259] Loaded cluster ID: 2a828ab342e346e2a5d5adf6e3b4bd84
I20260806 23:56:41.665691 12147 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525: loading cluster ID for follower catalog manager: success
I20260806 23:56:41.667392 12147 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525: acquiring CA information for follower catalog manager: success
I20260806 23:56:41.668602 12147 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ff6c53966f542c0aef50517c4bc8525: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:56:41.716253 12208 catalog_manager.cc:1259] Loaded cluster ID: 2a828ab342e346e2a5d5adf6e3b4bd84
I20260806 23:56:41.716347 12208 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd: loading cluster ID for follower catalog manager: success
I20260806 23:56:41.718010 12208 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd: acquiring CA information for follower catalog manager: success
I20260806 23:56:41.719168 12208 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99dd6e466b3346029e0d2e20697bbdbd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:56:41.913463 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:41.906106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.10.225.189:38237)
0806 23:56:41.906747 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:41.906751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:41.906785 (+ 34us) server_negotiation.cc:407] Connection header received
0806 23:56:41.906885 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:41.906887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:41.907138 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:41.907299 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:41.907968 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.908904 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.910609 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.911056 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.911193 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:41.911877 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:41.912750 (+ 873us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:41.912908 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:41.912991 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 23:56:41.913085 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":166,"threads_started":1}
I20260806 23:56:41.926658 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:41.919507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.10.225.188:33897)
0806 23:56:41.919858 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:41.919878 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:41.919928 (+ 50us) server_negotiation.cc:407] Connection header received
0806 23:56:41.920173 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:41.920178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:41.920394 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:41.920684 (+ 290us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:41.921120 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.922004 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.923513 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.924145 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.924304 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:41.924867 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:41.925967 (+ 1100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:41.926128 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:41.926318 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 23:56:41.926426 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":91,"threads_started":1}
I20260806 23:56:41.939100 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:41.932670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.10.225.189:38237)
0806 23:56:41.933024 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:41.933049 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:41.933431 (+ 382us) server_negotiation.cc:407] Connection header received
0806 23:56:41.933677 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:41.933689 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:41.933786 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:41.933928 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:41.934472 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.935795 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.937269 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.937604 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.937888 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:41.938316 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:41.938453 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:41.938558 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:41.938707 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 23:56:41.938792 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260806 23:56:41.943038 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:41.933848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.10.225.188:33897)
0806 23:56:41.934028 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:41.934044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:41.934579 (+ 535us) server_negotiation.cc:407] Connection header received
0806 23:56:41.934813 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:41.934817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:41.934990 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:41.935107 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:41.935731 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.937365 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.940205 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.940804 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.941059 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:41.941871 (+ 812us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:41.942070 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:41.942191 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:41.942433 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 23:56:41.942559 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 23:56:41.944324 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:41.932626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.10.225.190:35459)
0806 23:56:41.933339 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:41.933346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:41.934083 (+ 737us) server_negotiation.cc:407] Connection header received
0806 23:56:41.934446 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:41.934461 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:41.934866 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:41.935108 (+ 242us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:41.935956 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.937643 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.941150 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:41.941782 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:41.941996 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:41.942528 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:41.943176 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:41.943369 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:41.943885 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 23:56:41.943995 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":144,"threads_started":1}
I20260806 23:56:41.949884 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11804
I20260806 23:56:41.973533 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 11911
I20260806 23:56:41.998232 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 12017
2026-08-06T23:56:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3315 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 23:56:42 dist-test-slave-nj21 krb5kdc[12336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 23:56:42 dist-test-slave-nj21 krb5kdc[12336](info): set up 2 sockets
Aug 06 23:56:42 dist-test-slave-nj21 krb5kdc[12336](info): commencing operation
krb5kdc: starting...
W20260806 23:56:43.354929 11142 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.002s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/test-user.keytab.
Aug 06 23:56:43 dist-test-slave-nj21 krb5kdc[12336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T23:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T23:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.225.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
Entry for principal kudu/127.10.225.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab.
I20260806 23:56:43.492129 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:42253
--webserver_interface=127.10.225.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:42253,127.10.225.189:42037,127.10.225.188:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 23:56:43.607537 12361 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 23:56:43.607895 12361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:43.607944 12361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:43.608009 12361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:43.612076 12361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:43.612142 12361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:43.612169 12361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:43.612191 12361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:43.612212 12361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:43.617628 12361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:42253,127.10.225.189:42037,127.10.225.188:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.190.keytab
--principal=kudu/127.10.225.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.190:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.225.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:43.618744 12361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:43.619807 12361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:43.628851 12367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:43.628904 12368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:43.628993 12361 server_base.cc:1034] running on GCE node
W20260806 23:56:43.629164 12371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:43.629773 12361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:43.630928 12361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:43.632105 12361 hybrid_clock.cc:648] HybridClock initialized: now 1786060603632083 us; error 37 us; skew 500 ppm
Aug 06 23:56:43 dist-test-slave-nj21 krb5kdc[12336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:43.635144 12361 init.cc:377] Logged in from keytab as kudu/127.10.225.190@KRBTEST.COM (short username kudu)
I20260806 23:56:43.636126 12361 webserver.cc:460] Webserver started at http://127.10.225.190:35577/ using document root <none> and password file <none>
I20260806 23:56:43.636679 12361 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:43.636750 12361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:43.636976 12361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:43.639142 12361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/data/instance:
uuid: "4bb5b5ef76444446b8a211b71a95ee87"
format_stamp: "Formatted at 2026-08-06 23:56:43 on dist-test-slave-nj21"
server_key: "0e845890b6f8f4a71d212bb4946d8cc6"
server_key_iv: "9dff0f1ad91cc86f3f19df54f35e2de6"
server_key_version: "encryptionkey@0"
I20260806 23:56:43.639756 12361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/wal/instance:
uuid: "4bb5b5ef76444446b8a211b71a95ee87"
format_stamp: "Formatted at 2026-08-06 23:56:43 on dist-test-slave-nj21"
server_key: "0e845890b6f8f4a71d212bb4946d8cc6"
server_key_iv: "9dff0f1ad91cc86f3f19df54f35e2de6"
server_key_version: "encryptionkey@0"
I20260806 23:56:43.643544 12361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 23:56:43.646543 12381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:43.648242 12361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 23:56:43.648398 12361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/wal
uuid: "4bb5b5ef76444446b8a211b71a95ee87"
format_stamp: "Formatted at 2026-08-06 23:56:43 on dist-test-slave-nj21"
server_key: "0e845890b6f8f4a71d212bb4946d8cc6"
server_key_iv: "9dff0f1ad91cc86f3f19df54f35e2de6"
server_key_version: "encryptionkey@0"
I20260806 23:56:43.648514 12361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:43.671648 12361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:43.675001 12361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:43.675194 12361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:43.684360 12361 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.190:42253
I20260806 23:56:43.684631 12462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.190:42253 every 8 connection(s)
I20260806 23:56:43.685391 12361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/data/info.pb
I20260806 23:56:43.688370 12464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:43.690186 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 12361
I20260806 23:56:43.690312 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-0/wal/instance
I20260806 23:56:43.690625 11142 external_mini_cluster.cc:1428] Setting key 24ae72ba9cd2de8d370b019ebe47a6ec
I20260806 23:56:43.693137 12464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.225.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.189@KRBTEST.COM" created.
Aug 06 23:56:43 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:43.708638 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.695573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:42253 (local address 127.0.0.1:40852)
0806 23:56:43.696138 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.696189 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:43.697281 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:43.698031 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:43.698037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:43.698512 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:43.698882 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:43.698904 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.700130 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.700134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:43.701014 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:43.701025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.701700 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.701703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:43.701784 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:43.702324 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:43.702343 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:43.704004 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:43.705727 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:43.705736 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:43.705747 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:43.705969 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:43.706258 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:43.706261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:43.706263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:43.706373 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:43.706946 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:43.706953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:43.707088 (+ 135us) client_negotiation.cc:769] Sending connection context
0806 23:56:43.707237 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 23:56:43.707610 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":89,"thread_start_us":140,"threads_started":1}
I20260806 23:56:43.708638 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.695572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.10.225.190:42253)
0806 23:56:43.696186 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:43.696202 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:43.696301 (+ 99us) server_negotiation.cc:407] Connection header received
0806 23:56:43.697634 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:43.697639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:43.697831 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:43.697976 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:43.699040 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.700007 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.701140 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.701546 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.701671 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:43.704145 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:43.704150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:43.704161 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:43.704181 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:43.705629 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:43.706069 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:43.706074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:43.706075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:43.706177 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:43.706453 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:43.706456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:43.706458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:43.706840 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:43.706964 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:43.707314 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 23:56:43.707548 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":515,"spinlock_wait_cycles":37760,"thread_start_us":190,"threads_started":1}
I20260806 23:56:43.710995 12464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:43.711838 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.711517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:42037 (local address 127.0.0.1:46724)
0806 23:56:43.711659 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.711763 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 23:56:43.712049 12384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
I20260806 23:56:43.712992 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.712775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:42037 (local address 127.0.0.1:46736)
0806 23:56:43.712891 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.712937 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 23:56:43.713122 12464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.189:42037: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab.
Entry for principal kudu/127.10.225.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab.
I20260806 23:56:43.718986 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.189:42037
--webserver_interface=127.10.225.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab
--principal=kudu/127.10.225.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:42253,127.10.225.189:42037,127.10.225.188:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:56:43.762437 12464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } attempt: 1
I20260806 23:56:43.763707 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.763256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:42037 (local address 127.0.0.1:46738)
0806 23:56:43.763491 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.763567 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 23:56:43.764611 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.764375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:42037 (local address 127.0.0.1:46750)
0806 23:56:43.764496 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.764546 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 23:56:43.764765 12464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.189:42037: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
I20260806 23:56:43.832950 12464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } attempt: 2
I20260806 23:56:43.833819 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.833527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:42037 (local address 127.0.0.1:46754)
0806 23:56:43.833679 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.833739 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 23:56:43.834947 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.834676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:42037 (local address 127.0.0.1:46762)
0806 23:56:43.834812 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.834878 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 23:56:43.835192 12464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.189:42037: Network error: Client connection negotiation failed: client connection to 127.10.225.189:42037: connect: Connection refused (error 111)
W20260806 23:56:43.840147 12471 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 23:56:43.840713 12471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:43.840795 12471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:43.840866 12471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:43.846150 12471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:43.846232 12471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:43.846261 12471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:43.846285 12471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:43.846318 12471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:43.852375 12471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:42253,127.10.225.189:42037,127.10.225.188:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.189.keytab
--principal=kudu/127.10.225.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.189:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.225.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:43.853623 12471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:43.854977 12471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:43.863618 12479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:43.863741 12478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:43.863626 12489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:43.863855 12471 server_base.cc:1034] running on GCE node
I20260806 23:56:43.864511 12471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:43.865592 12471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:43.866765 12471 hybrid_clock.cc:648] HybridClock initialized: now 1786060603866729 us; error 65 us; skew 500 ppm
Aug 06 23:56:43 dist-test-slave-nj21 krb5kdc[12336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:43.873215 12471 init.cc:377] Logged in from keytab as kudu/127.10.225.189@KRBTEST.COM (short username kudu)
I20260806 23:56:43.874250 12471 webserver.cc:460] Webserver started at http://127.10.225.189:38819/ using document root <none> and password file <none>
I20260806 23:56:43.874837 12471 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:43.874912 12471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:43.875120 12471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:43.877426 12471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/data/instance:
uuid: "bf5f480b9338430487f0e9142236ee5b"
format_stamp: "Formatted at 2026-08-06 23:56:43 on dist-test-slave-nj21"
server_key: "301e0301182e5cc3f2e71d80f2fcc6ed"
server_key_iv: "36de95b5f5e8e4732cdd7820fe51bf97"
server_key_version: "encryptionkey@0"
I20260806 23:56:43.878011 12471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/wal/instance:
uuid: "bf5f480b9338430487f0e9142236ee5b"
format_stamp: "Formatted at 2026-08-06 23:56:43 on dist-test-slave-nj21"
server_key: "301e0301182e5cc3f2e71d80f2fcc6ed"
server_key_iv: "36de95b5f5e8e4732cdd7820fe51bf97"
server_key_version: "encryptionkey@0"
I20260806 23:56:43.882323 12471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 23:56:43.885216 12497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:43.886489 12471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 23:56:43.886611 12471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/wal
uuid: "bf5f480b9338430487f0e9142236ee5b"
format_stamp: "Formatted at 2026-08-06 23:56:43 on dist-test-slave-nj21"
server_key: "301e0301182e5cc3f2e71d80f2fcc6ed"
server_key_iv: "36de95b5f5e8e4732cdd7820fe51bf97"
server_key_version: "encryptionkey@0"
I20260806 23:56:43.886715 12471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:43.905642 12471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:43.908685 12471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:43.908880 12471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:43.915730 12471 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.189:42037
I20260806 23:56:43.916065 12569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.189:42037 every 8 connection(s)
I20260806 23:56:43.916733 12471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/data/info.pb
I20260806 23:56:43.919848 12573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:43.923926 12573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:43.926512 12464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } attempt: 3
I20260806 23:56:43.927022 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 12471
I20260806 23:56:43.927136 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-1/wal/instance
I20260806 23:56:43.927754 11142 external_mini_cluster.cc:1428] Setting key 1a34292b320476e9d8cd37aad8d6ecc7
Aug 06 23:56:43 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.225.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.225.188@KRBTEST.COM" created.
Aug 06 23:56:43 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:43.942996 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.926909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:42037 (local address 127.0.0.1:46776)
0806 23:56:43.927092 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.927106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:43.927204 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:43.929477 (+ 2273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:43.929480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:43.929681 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:43.929822 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:43.929830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.931993 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.931997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:43.932744 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:43.932756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.933961 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.933966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:43.934107 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:43.934651 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:43.934667 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:43.938372 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:43.940816 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:43.940819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:43.940821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:43.941053 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:43.941465 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:43.941468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:43.941470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:43.941574 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:43.942458 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:43.942462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:43.942584 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 23:56:43.942680 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 23:56:43.942853 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 23:56:43.945034 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.927465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.10.225.189:42037)
0806 23:56:43.928058 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:43.928078 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:43.928121 (+ 43us) server_negotiation.cc:407] Connection header received
0806 23:56:43.929036 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:43.929055 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:43.929280 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:43.929485 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:43.930121 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.931848 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.932888 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.933789 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.934476 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:43.938541 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:43.938549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:43.938565 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:43.938596 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:43.940643 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:43.941154 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:43.941160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:43.941181 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:43.941344 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:43.941691 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:43.941694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:43.941695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:43.942317 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:43.943322 (+ 1005us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:43.943511 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 23:56:43.943933 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":451,"thread_start_us":167,"threads_started":1}
I20260806 23:56:43.946025 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.926416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.10.225.190:42253)
0806 23:56:43.926582 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:43.926587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:43.928297 (+ 1710us) server_negotiation.cc:407] Connection header received
0806 23:56:43.929292 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:43.929295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:43.929345 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:43.929419 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:43.932892 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.933603 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.936432 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.937369 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.937489 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:43.942286 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:43.942290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:43.942293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:43.942319 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:43.944154 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:43.944687 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:43.944690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:43.944692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:43.944774 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:43.945056 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:43.945059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:43.945061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:43.945265 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:43.945351 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:43.945767 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 23:56:43.945879 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 23:56:43.946040 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.926727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:42253 (local address 127.0.0.1:40866)
0806 23:56:43.928140 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.928195 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:43.929053 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:43.929601 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:43.929606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:43.930061 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:43.931842 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:43.931849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.935192 (+ 3343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.935196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:43.936291 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:43.936304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.937676 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.937680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:43.937796 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:43.938400 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:43.938424 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:43.942126 (+ 3702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:43.944265 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:43.944276 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:43.944283 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:43.944587 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:43.944849 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:43.944865 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:43.944868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:43.944962 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:43.945406 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:43.945414 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:43.945536 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 23:56:43.945665 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 23:56:43.945853 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219,"mutex_wait_us":1554,"thread_start_us":137,"threads_started":1}
I20260806 23:56:43.947299 12573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:43.947492 12464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:43.948167 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.947898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:32926)
0806 23:56:43.948046 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.948095 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 23:56:43.948969 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.948680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:32930)
0806 23:56:43.948857 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.948907 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260806 23:56:43.949143 12464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:39075: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Aug 06 23:56:43 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.225.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab.
Entry for principal kudu/127.10.225.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab.
I20260806 23:56:43.960073 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.947826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:42037 (local address 127.0.0.1:46778)
0806 23:56:43.948029 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.948043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:43.948116 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:43.948476 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:43.948479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:43.948652 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:43.948778 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:43.948787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.951977 (+ 3190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.951981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:43.953070 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:43.953084 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.953927 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.953930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:43.954024 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:43.954501 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:43.954517 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:43.956396 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:43.958430 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:43.958434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:43.958436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:43.958698 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:43.959019 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:43.959021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:43.959023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:43.959108 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:43.959526 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:43.959529 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:43.959611 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 23:56:43.959724 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 23:56:43.959879 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260806 23:56:43.960526 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.947992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.10.225.189:42037)
0806 23:56:43.948095 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:43.948100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:43.948114 (+ 14us) server_negotiation.cc:407] Connection header received
0806 23:56:43.948310 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:43.948314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:43.948369 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:43.948451 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:43.948884 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.950014 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.953328 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:43.953669 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:43.953786 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:43.956539 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:43.956543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:43.956546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:43.956575 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:43.958317 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:43.958783 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:43.958800 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:43.958802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:43.958920 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:43.959178 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:43.959182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:43.959183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:43.959405 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:43.959495 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:43.960284 (+ 789us) server_negotiation.cc:299] Negotiation successful
0806 23:56:43.960397 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 23:56:43.960630 11142 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyweHpq/build/debug/bin/kudu
/tmp/dist-test-taskyweHpq/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.188:39075
--webserver_interface=127.10.225.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab
--principal=kudu/127.10.225.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.225.148:41365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.225.190:42253,127.10.225.189:42037,127.10.225.188:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 23:56:43.963349 12573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:43.964210 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.963949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:32938)
0806 23:56:43.964054 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.964144 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 23:56:43.964847 12498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
I20260806 23:56:43.965855 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.965587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:32952)
0806 23:56:43.965721 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.965774 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 23:56:43.966037 12573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:39075: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
I20260806 23:56:43.980289 12464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } attempt: 1
I20260806 23:56:43.981187 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.980871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:32958)
0806 23:56:43.981019 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.981084 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 23:56:43.981868 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:43.981638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:32968)
0806 23:56:43.981735 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:43.981786 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 23:56:43.982014 12464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:39075: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
I20260806 23:56:44.015349 12573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } attempt: 1
I20260806 23:56:44.016343 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.015995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:32980)
0806 23:56:44.016171 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.016246 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 23:56:44.017118 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.016903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:32986)
0806 23:56:44.017012 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.017058 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 23:56:44.017283 12573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:39075: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
I20260806 23:56:44.057184 12464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } attempt: 2
I20260806 23:56:44.058112 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.057797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:32992)
0806 23:56:44.057959 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.058020 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 23:56:44.058796 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.058584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:33000)
0806 23:56:44.058684 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.058729 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 23:56:44.058939 12464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:39075: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
W20260806 23:56:44.080361 12583 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 23:56:44.080660 12583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 23:56:44.080705 12583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 23:56:44.080772 12583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 23:56:44.084851 12583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 23:56:44.084915 12583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 23:56:44.084944 12583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 23:56:44.084966 12583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 23:56:44.084995 12583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 23:56:44.085448 12573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } attempt: 2
I20260806 23:56:44.086333 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.086019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:33002)
0806 23:56:44.086182 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.086252 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 23:56:44.087101 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.086874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:33008)
0806 23:56:44.086998 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.087043 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 23:56:44.087248 12573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:39075: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
I20260806 23:56:44.090809 12583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.225.148:41365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.225.190:42253,127.10.225.189:42037,127.10.225.188:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/krb5kdc/kudu_127.10.225.188.keytab
--principal=kudu/127.10.225.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.225.188:39075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.225.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 23:05:08 UTC on ccycloud.quasar-eswqwf.root.comops.site
build id 4571539
I20260806 23:56:44.091991 12583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 23:56:44.093017 12583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 23:56:44.101557 12591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 23:56:44.101568 12595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:44.101800 12583 server_base.cc:1034] running on GCE node
W20260806 23:56:44.102272 12590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 23:56:44.102803 12583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 23:56:44.103971 12583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 23:56:44.105154 12583 hybrid_clock.cc:648] HybridClock initialized: now 1786060604105136 us; error 27 us; skew 500 ppm
Aug 06 23:56:44 dist-test-slave-nj21 krb5kdc[12336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060604, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 23:56:44.107908 12583 init.cc:377] Logged in from keytab as kudu/127.10.225.188@KRBTEST.COM (short username kudu)
I20260806 23:56:44.108800 12583 webserver.cc:460] Webserver started at http://127.10.225.188:38283/ using document root <none> and password file <none>
I20260806 23:56:44.109330 12583 fs_manager.cc:359] Metadata directory not provided
I20260806 23:56:44.109387 12583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 23:56:44.109575 12583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 23:56:44.111613 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/data/instance:
uuid: "ea821937a3914151ae13b85080e43eef"
format_stamp: "Formatted at 2026-08-06 23:56:44 on dist-test-slave-nj21"
server_key: "d6821941207ed9761117e6610605c0e7"
server_key_iv: "ab6ae4447afca975a2650b3c247c6864"
server_key_version: "encryptionkey@0"
I20260806 23:56:44.112170 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/wal/instance:
uuid: "ea821937a3914151ae13b85080e43eef"
format_stamp: "Formatted at 2026-08-06 23:56:44 on dist-test-slave-nj21"
server_key: "d6821941207ed9761117e6610605c0e7"
server_key_iv: "ab6ae4447afca975a2650b3c247c6864"
server_key_version: "encryptionkey@0"
I20260806 23:56:44.115918 12583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 23:56:44.118341 12608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 23:56:44.119715 12583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 23:56:44.119869 12583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/data,/tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/wal
uuid: "ea821937a3914151ae13b85080e43eef"
format_stamp: "Formatted at 2026-08-06 23:56:44 on dist-test-slave-nj21"
server_key: "d6821941207ed9761117e6610605c0e7"
server_key_iv: "ab6ae4447afca975a2650b3c247c6864"
server_key_version: "encryptionkey@0"
I20260806 23:56:44.119976 12583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 23:56:44.152103 12583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 23:56:44.155097 12583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 23:56:44.155300 12583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 23:56:44.158102 12464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } attempt: 3
I20260806 23:56:44.158967 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.158628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:33010)
0806 23:56:44.158781 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.158856 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 23:56:44.159874 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.159625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:33024)
0806 23:56:44.159758 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.159806 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 23:56:44.159989 12464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.225.188:39075: Network error: Client connection negotiation failed: client connection to 127.10.225.188:39075: connect: Connection refused (error 111)
I20260806 23:56:44.163401 12583 rpc_server.cc:307] RPC server started. Bound to: 127.10.225.188:39075
I20260806 23:56:44.163704 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.225.188:39075 every 8 connection(s)
I20260806 23:56:44.164422 12583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/data/info.pb
I20260806 23:56:44.167173 12684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 23:56:44.169535 11142 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyweHpq/build/debug/bin/kudu as pid 12583
I20260806 23:56:44.169658 11142 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyweHpq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786060299689793-11142-0/minicluster-data/master-2/wal/instance
I20260806 23:56:44.169992 11142 external_mini_cluster.cc:1428] Setting key fca8336b0a54f35c3b3dcc4b2c2feacd
I20260806 23:56:44.171502 12684 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:44.178409 12573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } attempt: 3
Aug 06 23:56:44 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060604, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
Aug 06 23:56:44 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
Aug 06 23:56:44 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.189@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
I20260806 23:56:44.184244 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.173249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.10.225.190:42253)
0806 23:56:44.173385 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.173390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.173785 (+ 395us) server_negotiation.cc:407] Connection header received
0806 23:56:44.173932 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.173935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.173980 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.174037 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.174741 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.175469 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.176812 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.177132 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.177245 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.179991 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.179995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.179998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.180026 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.181838 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.182462 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.182466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.182468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.182553 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.182982 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.182986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.182988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.183223 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.183328 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.183948 (+ 620us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.184076 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:56:44.185294 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.173187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.190:42253 (local address 127.0.0.1:40872)
0806 23:56:44.173638 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.173690 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:44.173825 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:44.174116 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:44.174121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:44.174445 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:44.174600 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.174609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.175581 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.175585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.176690 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.176700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.177272 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.177276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.177399 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.177968 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:44.177988 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:44.179855 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:44.181977 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.181986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.181997 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.182340 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.182671 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.182674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.182676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.182879 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.183337 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:44.183343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:44.183654 (+ 311us) client_negotiation.cc:769] Sending connection context
0806 23:56:44.183860 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 23:56:44.184203 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":4,"spinlock_wait_cycles":4736,"thread_start_us":164,"threads_started":1}
I20260806 23:56:44.186095 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.171402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.10.225.188:39075)
0806 23:56:44.171879 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.171899 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.171944 (+ 45us) server_negotiation.cc:407] Connection header received
0806 23:56:44.172600 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.172616 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.172992 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.173323 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.174160 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.175420 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.176641 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.177131 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.177303 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.180330 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.180335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.180345 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.180367 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.182281 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.182761 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.182767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.182768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.182879 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.183164 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.183167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.183169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.183653 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.184769 (+ 1116us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.184887 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.185054 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":327,"spinlock_wait_cycles":5632,"thread_start_us":124,"threads_started":1}
I20260806 23:56:44.186877 12684 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:44.187819 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.178888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:33032)
0806 23:56:44.179008 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.179021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:44.179093 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:44.179480 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:44.179484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:44.179639 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:44.179771 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.179780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.181144 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.181148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.181732 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.181746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.182364 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.182369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.182461 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.182905 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:44.182917 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:44.184196 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:44.186189 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.186192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.186195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.186445 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.186705 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.186709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.186711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.186790 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.187264 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:44.187268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:44.187381 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 23:56:44.187485 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 23:56:44.187634 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 23:56:44.188313 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.178958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.10.225.188:39075)
0806 23:56:44.179252 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.179259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.179269 (+ 10us) server_negotiation.cc:407] Connection header received
0806 23:56:44.179301 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.179303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.179344 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.179418 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.179899 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.181025 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.181885 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.182224 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.182671 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.184431 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.184435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.184438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.184461 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.185924 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.186541 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.186544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.186546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.186624 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.186867 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.186870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.186872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.187171 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.187265 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.188019 (+ 754us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.188154 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":95,"threads_started":1}
I20260806 23:56:44.191342 12573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b: Bootstrap starting.
Aug 06 23:56:44 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.190@KRBTEST.COM
I20260806 23:56:44.194017 12573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:44.194746 12573 log.cc:824] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:44.197204 12573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b: No bootstrap required, opened a new log
I20260806 23:56:44.197391 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.188461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.10.225.190:42253)
0806 23:56:44.188559 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.188563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.188756 (+ 193us) server_negotiation.cc:407] Connection header received
0806 23:56:44.188853 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.188856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.188907 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.188969 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.189648 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.190211 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.190983 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.191203 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.191288 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.193754 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.193757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.193759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.193782 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.195179 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.195630 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.195634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.195636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.195750 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.196034 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.196037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.196039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.196247 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.196473 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.197102 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.197225 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Aug 06 23:56:44 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060604, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
I20260806 23:56:44.201545 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.187390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.189:42037 (local address 127.0.0.1:46788)
0806 23:56:44.187567 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.187580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:44.187647 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:44.188004 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:44.188007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:44.188181 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:44.188271 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.188278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.195165 (+ 6887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.195168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.195956 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.195971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.196470 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.196474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.196565 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.196974 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:44.196989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:44.198284 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:44.200052 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.200056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.200058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.200249 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.200623 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.200625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.200626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.200699 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.201054 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:44.201057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:44.201150 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 23:56:44.201236 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 23:56:44.201375 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260806 23:56:44.201552 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.187481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.10.225.189:42037)
0806 23:56:44.187590 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.187594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.187609 (+ 15us) server_negotiation.cc:407] Connection header received
0806 23:56:44.187777 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.187780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.187823 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.187871 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.188413 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.189015 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.196118 (+ 7103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.196355 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.196444 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.198426 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.198430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.198433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.198461 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.199952 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.200414 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.200417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.200419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.200526 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.200786 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.200790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.200792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.200963 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.201050 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.201300 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.201422 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 23:56:44.202028 12573 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } }
I20260806 23:56:44.202286 12573 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5f480b9338430487f0e9142236ee5b, State: Initialized, Role: FOLLOWER
I20260806 23:56:44.202286 12684 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } has no permanent_uuid. Determining permanent_uuid...
I20260806 23:56:44.202634 12573 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } }
I20260806 23:56:44.203176 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.203334 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:44.204206 12573 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 23:56:44 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.225.189@KRBTEST.COM
Aug 06 23:56:44 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060604, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.188@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
I20260806 23:56:44.209430 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.198611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.10.225.189:42037)
0806 23:56:44.199149 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.199157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.199170 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:56:44.199199 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.199202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.199242 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.200161 (+ 919us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.200256 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.201113 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.202132 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.202464 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.203737 (+ 1273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.205438 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.205442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.205445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.205469 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.207049 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.207502 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.207506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.207508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.207606 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.207962 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.207965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.207967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.208176 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.209014 (+ 838us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.209098 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.209225 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":345,"threads_started":1}
W20260806 23:56:44.210768 12711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:44.210852 12711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:44.211167 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.202784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:33048)
0806 23:56:44.202951 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.202963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:44.203042 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:44.203626 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:44.203628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:44.203944 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:44.204035 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.204041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.204983 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.204985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.205734 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.205746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.206394 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.206398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.206461 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.206893 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:44.206906 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:44.208242 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:44.209932 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.209934 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.209935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.210082 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.210317 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.210320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.210321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.210399 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.210764 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:44.210767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:44.210850 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 23:56:44.210945 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 23:56:44.211056 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 23:56:44.211613 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.202957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.10.225.188:39075)
0806 23:56:44.203355 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.203370 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.203383 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:56:44.203417 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.203419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.203461 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.203531 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.204147 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.204866 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.205859 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.206186 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.206280 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.208351 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.208355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.208369 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.208390 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.209845 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.210166 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.210169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.210171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.210240 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.210474 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.210477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.210479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.210678 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.210762 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.211421 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.211517 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325}
I20260806 23:56:44.208094 12471 master_runner.cc:186] Error getting master registration for 127.10.225.190:42253: OK, instance_id { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" instance_seqno: 1786060603681925 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 23:56:44.211871 12471 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:56:44.214504 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef: Bootstrap starting.
I20260806 23:56:44.216292 11142 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 23:56:44.217128 12684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:44.218067 12684 log.cc:824] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:44.217414 12423 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4bb5b5ef76444446b8a211b71a95ee87"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40884
I20260806 23:56:44.220548 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef: No bootstrap required, opened a new log
I20260806 23:56:44.224895 12684 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } }
I20260806 23:56:44.225174 12684 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea821937a3914151ae13b85080e43eef, State: Initialized, Role: FOLLOWER
I20260806 23:56:44.225548 12684 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } }
I20260806 23:56:44.226081 12716 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.226222 12716 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:44.227051 12684 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [sys.catalog]: configured and running, proceeding with master startup.
W20260806 23:56:44.232731 12727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:44.232820 12727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:44.230868 12583 master_runner.cc:186] Error getting master registration for 127.10.225.190:42253: OK, instance_id { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" instance_seqno: 1786060603681925 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 23:56:44.234191 12583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:56:44.324095 12464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } attempt: 4
Aug 06 23:56:44 dist-test-slave-nj21 krb5kdc[12336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786060603, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.225.190@KRBTEST.COM for kudu/127.10.225.188@KRBTEST.COM
I20260806 23:56:44.334393 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.324740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.225.188:39075 (local address 127.0.0.1:33058)
0806 23:56:44.324995 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 23:56:44.325012 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 23:56:44.325120 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 23:56:44.325427 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 23:56:44.325430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 23:56:44.325646 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 23:56:44.325763 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.325770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.326758 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.326762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.327623 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 23:56:44.327636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.328229 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.328233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 23:56:44.328315 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.328899 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 23:56:44.328914 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 23:56:44.330675 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 23:56:44.332664 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.332668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.332670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.332923 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.333218 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 23:56:44.333222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 23:56:44.333224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 23:56:44.333322 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 23:56:44.333837 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 23:56:44.333840 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 23:56:44.333948 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 23:56:44.334047 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 23:56:44.334199 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260806 23:56:44.335060 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.324804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.10.225.188:39075)
0806 23:56:44.324946 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.324952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.325068 (+ 116us) server_negotiation.cc:407] Connection header received
0806 23:56:44.325220 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.325223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.325281 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.325392 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.325898 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.326648 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.327763 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.328108 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.328239 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.330843 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.330848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.330851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.330881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.332556 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.333026 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.333029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.333031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.333120 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.333448 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.333452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.333453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.333695 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.333790 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.334787 (+ 997us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.334917 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 23:56:44.338989 12464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87: Bootstrap starting.
I20260806 23:56:44.341993 12464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87: Neither blocks nor log segments found. Creating new log.
I20260806 23:56:44.342938 12464 log.cc:824] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87: Log is configured to *not* fsync() on all Append() calls
I20260806 23:56:44.345438 12464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87: No bootstrap required, opened a new log
I20260806 23:56:44.348790 12464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } }
I20260806 23:56:44.349047 12464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb5b5ef76444446b8a211b71a95ee87, State: Initialized, Role: FOLLOWER
I20260806 23:56:44.349431 12464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } }
I20260806 23:56:44.350099 12732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.350387 12732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:44.351011 12464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 23:56:44.351073 12423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 23:56:44.351191 12423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:44.352641 12423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } }
I20260806 23:56:44.354725 12534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb5b5ef76444446b8a211b71a95ee87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bf5f480b9338430487f0e9142236ee5b"
I20260806 23:56:44.354979 12534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:44.356441 12534 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb5b5ef76444446b8a211b71a95ee87 in term 1.
I20260806 23:56:44.356940 12650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bb5b5ef76444446b8a211b71a95ee87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ea821937a3914151ae13b85080e43eef"
I20260806 23:56:44.357182 12650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [term 0 FOLLOWER]: Advancing to term 1
I20260806 23:56:44.358592 12650 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb5b5ef76444446b8a211b71a95ee87 in term 1.
I20260806 23:56:44.359382 12383 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb5b5ef76444446b8a211b71a95ee87, ea821937a3914151ae13b85080e43eef; no voters:
I20260806 23:56:44.353644 12423 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [CANDIDATE]: Term 1 election: Requested vote from peers bf5f480b9338430487f0e9142236ee5b (127.10.225.189:42037), ea821937a3914151ae13b85080e43eef (127.10.225.188:39075)
W20260806 23:56:44.357484 12748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 23:56:44.359891 12748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 23:56:44.368410 12361 master_runner.cc:263] No leader master found from master 4bb5b5ef76444446b8a211b71a95ee87
I20260806 23:56:44.368494 12361 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 23:56:44.369004 12732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 23:56:44.369338 12732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [term 1 LEADER]: Becoming Leader. State: Replica: 4bb5b5ef76444446b8a211b71a95ee87, State: Running, Role: LEADER
I20260806 23:56:44.369807 12732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } }
I20260806 23:56:44.371933 12732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb5b5ef76444446b8a211b71a95ee87. Latest consensus state: current_term: 1 leader_uuid: "4bb5b5ef76444446b8a211b71a95ee87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.372758 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.362553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.10.225.190:42253)
0806 23:56:44.362669 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.362674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.363951 (+ 1277us) server_negotiation.cc:407] Connection header received
0806 23:56:44.364117 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.364120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.364164 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.364209 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.365154 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.365923 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.367049 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.367408 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.367476 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.369345 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.369350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.369353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.369383 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.370945 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.371428 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.371432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.371434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.371515 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.371820 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.371824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.371825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.372041 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.372135 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.372485 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.372606 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 23:56:44.373943 12732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:44.380786 12766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 23:56:44.381839 12766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 23:56:44.382284 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.363369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.10.225.188:39075)
0806 23:56:44.363492 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.363496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.373280 (+ 9784us) server_negotiation.cc:407] Connection header received
0806 23:56:44.373470 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.373473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.373529 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.373577 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.374581 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.375378 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.376466 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.376801 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.376876 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.378551 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.378556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.378559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.378591 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.380247 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.380827 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.380831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.380832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.380928 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.381248 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.381252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.381254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.381502 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.381635 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.381990 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.382114 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:56:44.387367 12650 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [term 1 FOLLOWER]: Refusing update from remote peer 4bb5b5ef76444446b8a211b71a95ee87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:44.388453 12534 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [term 1 FOLLOWER]: Refusing update from remote peer 4bb5b5ef76444446b8a211b71a95ee87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 23:56:44.389096 12732 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:44.389530 12732 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 23:56:44.393596 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb5b5ef76444446b8a211b71a95ee87. Latest consensus state: current_term: 1 leader_uuid: "4bb5b5ef76444446b8a211b71a95ee87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.393729 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:44.394289 12716 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb5b5ef76444446b8a211b71a95ee87. Latest consensus state: current_term: 1 leader_uuid: "4bb5b5ef76444446b8a211b71a95ee87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.394425 12716 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:44.397131 12762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bb5b5ef76444446b8a211b71a95ee87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.397282 12762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:44.398757 12716 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bb5b5ef76444446b8a211b71a95ee87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.398890 12716 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:44.399266 12762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bb5b5ef76444446b8a211b71a95ee87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.399397 12762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87 [sys.catalog]: This master's current role is: LEADER
I20260806 23:56:44.399525 12767 mvcc.cc:205] Tried to move back new op lower bound from 7315704235563274240 to 7315704235503136768. Current Snapshot: MvccSnapshot[applied={T|T < 7315704235563274240}]
I20260806 23:56:44.399852 12768 mvcc.cc:205] Tried to move back new op lower bound from 7315704235563274240 to 7315704235503136768. Current Snapshot: MvccSnapshot[applied={T|T < 7315704235563274240}]
I20260806 23:56:44.400045 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:44.372470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.10.225.189:42037)
0806 23:56:44.372602 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.372607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.375949 (+ 3342us) server_negotiation.cc:407] Connection header received
0806 23:56:44.376127 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.376131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.376183 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.376231 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.377158 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.377918 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.379054 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:44.379417 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:44.379496 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:44.396117 (+ 16621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 23:56:44.396121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 23:56:44.396124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 23:56:44.396159 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 23:56:44.397963 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.398531 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.398535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.398537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.398635 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 23:56:44.398971 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 23:56:44.398974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 23:56:44.398976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 23:56:44.399220 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 23:56:44.399326 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:44.399734 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 23:56:44.399890 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 23:56:44.400060 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bb5b5ef76444446b8a211b71a95ee87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5b5ef76444446b8a211b71a95ee87" member_type: VOTER last_known_addr { host: "127.10.225.190" port: 42253 } } peers { permanent_uuid: "bf5f480b9338430487f0e9142236ee5b" member_type: VOTER last_known_addr { host: "127.10.225.189" port: 42037 } } peers { permanent_uuid: "ea821937a3914151ae13b85080e43eef" member_type: VOTER last_known_addr { host: "127.10.225.188" port: 39075 } } }
I20260806 23:56:44.400183 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 23:56:44.402565 12770 mvcc.cc:205] Tried to move back new op lower bound from 7315704235563274240 to 7315704235503136768. Current Snapshot: MvccSnapshot[applied={T|T < 7315704235563274240}]
I20260806 23:56:44.402807 12766 catalog_manager.cc:1347] Generated new cluster ID: 856045a760924194bac3a327a07e9751
I20260806 23:56:44.402880 12766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 23:56:44.418020 12766 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 23:56:44.419324 12766 catalog_manager.cc:1504] Loading token signing keys...
I20260806 23:56:44.434907 12766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bb5b5ef76444446b8a211b71a95ee87: Generated new TSK 0
I20260806 23:56:44.435891 12766 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 23:56:44.458884 12468 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.457033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.10.225.190:42253)
0806 23:56:44.457167 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.457172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.457836 (+ 664us) server_negotiation.cc:407] Connection header received
0806 23:56:44.458019 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.458022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.458073 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.458123 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.458735 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40906: BlockingRecv error: recv got EOF from 127.0.0.1:40906 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 23:56:44.458891 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.457433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.10.225.189:42037)
0806 23:56:44.457558 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.457562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.457673 (+ 111us) server_negotiation.cc:407] Connection header received
0806 23:56:44.457887 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.457891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.457945 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.457997 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.458756 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46832: BlockingRecv error: recv got EOF from 127.0.0.1:46832 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 23:56:44.458892 12692 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.457473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.10.225.188:39075)
0806 23:56:44.457592 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.457597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.457747 (+ 150us) server_negotiation.cc:407] Connection header received
0806 23:56:44.457925 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.457928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.457977 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.458026 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.458759 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33080: BlockingRecv error: recv got EOF from 127.0.0.1:33080 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 23:56:44.462373 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.461265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.10.225.189:42037)
0806 23:56:44.461377 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.461381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.461575 (+ 194us) server_negotiation.cc:407] Connection header received
0806 23:56:44.461723 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.461725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.461795 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.461861 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.462248 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46844: BlockingRecv error: recv got EOF from 127.0.0.1:46844 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260806 23:56:44.466213 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.465356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.10.225.189:42037)
0806 23:56:44.465469 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.465472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.465483 (+ 11us) server_negotiation.cc:407] Connection header received
0806 23:56:44.465523 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.465525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.465564 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.465629 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.466105 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46850: BlockingRecv error: recv got EOF from 127.0.0.1:46850 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 23:56:44.470804 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.469901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46866 (local address 127.10.225.189:42037)
0806 23:56:44.470008 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.470012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.470126 (+ 114us) server_negotiation.cc:407] Connection header received
0806 23:56:44.470223 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.470226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.470274 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.470363 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.470699 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46866: BlockingRecv error: recv got EOF from 127.0.0.1:46866 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 23:56:44.477600 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.476575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.10.225.189:42037)
0806 23:56:44.476724 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.476728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.476839 (+ 111us) server_negotiation.cc:407] Connection header received
0806 23:56:44.476966 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.476970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.477017 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.477082 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.477489 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46878: BlockingRecv error: recv got EOF from 127.0.0.1:46878 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260806 23:56:44.488615 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.487530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.10.225.189:42037)
0806 23:56:44.487664 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.487689 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.487790 (+ 101us) server_negotiation.cc:407] Connection header received
0806 23:56:44.487940 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.487942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.487991 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.488058 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.488497 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46886: BlockingRecv error: recv got EOF from 127.0.0.1:46886 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 23:56:44.508157 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.506996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.10.225.189:42037)
0806 23:56:44.507164 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.507170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.507280 (+ 110us) server_negotiation.cc:407] Connection header received
0806 23:56:44.507398 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.507401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.507461 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.507566 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.508028 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46890: BlockingRecv error: recv got EOF from 127.0.0.1:46890 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 23:56:44.543315 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.542250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.10.225.189:42037)
0806 23:56:44.542403 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.542408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.542486 (+ 78us) server_negotiation.cc:407] Connection header received
0806 23:56:44.542607 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.542610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.542669 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.542769 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.543180 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46902: BlockingRecv error: recv got EOF from 127.0.0.1:46902 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 23:56:44.610754 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.609701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.10.225.189:42037)
0806 23:56:44.609821 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.609826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.609910 (+ 84us) server_negotiation.cc:407] Connection header received
0806 23:56:44.610035 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.610038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.610086 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.610164 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.610650 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46916: BlockingRecv error: recv got EOF from 127.0.0.1:46916 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 23:56:44.742223 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:44.741120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.10.225.189:42037)
0806 23:56:44.741264 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:44.741269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:44.741344 (+ 75us) server_negotiation.cc:407] Connection header received
0806 23:56:44.741517 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:44.741520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:44.741570 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:44.741640 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:44.742119 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43376: BlockingRecv error: recv got EOF from 127.0.0.1:43376 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 23:56:45.001655 12694 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 23:56:45.000672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.10.225.189:42037)
0806 23:56:45.000855 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:45.000860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:45.000879 (+ 19us) server_negotiation.cc:407] Connection header received
0806 23:56:45.000941 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:45.000944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:45.000993 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:45.001085 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 23:56:45.001530 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43386: BlockingRecv error: recv got EOF from 127.0.0.1:43386 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 23:56:45.212261 12711 catalog_manager.cc:1259] Loaded cluster ID: 856045a760924194bac3a327a07e9751
I20260806 23:56:45.212360 12711 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b: loading cluster ID for follower catalog manager: success
I20260806 23:56:45.214062 12711 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b: acquiring CA information for follower catalog manager: success
I20260806 23:56:45.215256 12711 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf5f480b9338430487f0e9142236ee5b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:56:45.234287 12727 catalog_manager.cc:1259] Loaded cluster ID: 856045a760924194bac3a327a07e9751
I20260806 23:56:45.234385 12727 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef: loading cluster ID for follower catalog manager: success
I20260806 23:56:45.236219 12727 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef: acquiring CA information for follower catalog manager: success
I20260806 23:56:45.237459 12727 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea821937a3914151ae13b85080e43eef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 23:56:45.519625 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:45.515884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.10.225.189:42037)
0806 23:56:45.516142 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:45.516147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:45.516164 (+ 17us) server_negotiation.cc:407] Connection header received
0806 23:56:45.516315 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:45.516317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:45.516458 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:45.516556 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:45.516813 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.517337 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.518188 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.518420 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.518506 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:45.518803 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:45.519237 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:45.519346 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:45.519410 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 23:56:45.519470 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260806 23:56:45.526304 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:45.522750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.10.225.188:39075)
0806 23:56:45.522993 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:45.522997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:45.523010 (+ 13us) server_negotiation.cc:407] Connection header received
0806 23:56:45.523114 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:45.523116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:45.523229 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:45.523323 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:45.523548 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.524113 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.524931 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.525156 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.525229 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:45.525477 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:45.525903 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:45.525989 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:45.526094 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 23:56:45.526142 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260806 23:56:45.533715 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:45.529656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.10.225.189:42037)
0806 23:56:45.529820 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:45.529825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:45.530176 (+ 351us) server_negotiation.cc:407] Connection header received
0806 23:56:45.530326 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:45.530329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:45.530386 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:45.530454 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:45.530785 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.531324 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.532579 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.532910 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.532990 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:45.533263 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:45.533346 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:45.533404 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:45.533512 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 23:56:45.533571 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 23:56:45.534067 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:45.529559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.10.225.190:42253)
0806 23:56:45.529820 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:45.529825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:45.530127 (+ 302us) server_negotiation.cc:407] Connection header received
0806 23:56:45.530325 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:45.530329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:45.530509 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:45.530636 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:45.530857 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.531632 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.532447 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.532744 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.532837 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:45.533123 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:45.533469 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:45.533548 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:45.533843 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 23:56:45.533916 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260806 23:56:45.534798 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 23:56:45.529886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.10.225.188:39075)
0806 23:56:45.530039 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 23:56:45.530043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 23:56:45.530294 (+ 251us) server_negotiation.cc:407] Connection header received
0806 23:56:45.530342 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 23:56:45.530344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 23:56:45.530385 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 23:56:45.530446 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 23:56:45.530677 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.531472 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.533060 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 23:56:45.533350 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 23:56:45.533461 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 23:56:45.533892 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 23:56:45.533968 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 23:56:45.534027 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 23:56:45.534476 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 23:56:45.534639 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 23:56:45.541664 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 12361
I20260806 23:56:45.553153 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 12471
I20260806 23:56:45.567916 11142 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyweHpq/build/debug/bin/kudu with pid 12583
2026-08-06T23:56:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3557 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6873 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (305845 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST